sia.hackernoon.com

In systems that need to handle large volumes of background data — such as billing routines — it’s common to fall into the trap of writing temporary scripts or running manual processes. That’s exactly what we set out to avoid.

In this article, I share how Oban helped us structure a resilient and scalable job system, becoming a core part of our billing process at Nextcode.

The challenge: complex and recurring billing routines

Our scenario involved:

Processing thousands of usage logs daily
Fetching logs from different apps and databases
Applying client-specific and service-specific rules
Aggregating and generating logs
Ensuring safe retries in case of failures
Horizontal scaling without losing traceability
Storing data in optimized databases for querying
Running these routines manually or with one-off scripts was risky, especially with no logging or fallback in case of failure

That’s when Oban came into play.

Why we chose Oban

Besides being built in pure Elixir, Oban gave us everything we needed:

✅ PostgreSQL persistence

✅ Automatic retries with backoff

✅ Job deduplication with uniqueness control

✅ Dashboard support via oban_web (yet to try it)

✅ Distributed execution with isolated queues and concurrency

✅ Native Ecto integration and flexibility

Installation

Getting started is simple. Just add the dependency to your mix.exs:

def deps do
  [
    {:oban, "~> 2.17"},
  ]
end

Then, configure your repo and supervision tree:

# config/config.exs
config :my_app, Oban,
  repo: MyApp.Repo,
  plugins: [
    {Oban.Plugins.Pruner, max_age: 86_400},
  ],
  queues: [
    mongodb_daily_log: 1
  ]

# application.ex
children = [
  {Oban, Application.fetch_env!(:my_app, Oban)}
]

Our worker: `MongodbDailyLog`

Here’s how we structured one of our workers to process daily logs for billing:

defmodule MyApp.Job.MongodbDailyLog do
  use Oban.Worker,
    queue: :mongodb_daily_log,
    max_attempts: 2,
    unique: [
      fields: [:args],
      states: [:available, :scheduled, :executing],
      period: 60
    ]

What does this do?

Assigns a specific queue for the job
Limits each job to 2 attempts
Enforces uniqueness to avoid duplicate executions with the same arguments

Automatic scheduling and safe execution

Our job can be triggered either automatically or on-demand. We use Timex to handle dates and define the processing window:

def perform(%Oban.Job{args: %{"date" => %{"day" => d, "month" => m, "year" => y}, "only_logs" => only_logs}}) do
  date = Timex.to_date({y, m, d})
  gte = Timex.to_datetime(date, "America/Sao_Paulo")
  lt = Timex.shift(gte, days: 1)

  job_impl().run(%{gte: gte, lt: lt}, only_logs)
end

Default execution for previous day if no date is specified:

def perform(%Oban.Job{args: %{}}) do
  %{day: d, month: m, year: y} = Timex.shift(Timex.today(), days: -1)
  %Oban.Job{args: %{"date" => %{"day" => d, "month" => m, "year" => y}, "only_logs" => false}}
  |> perform()
end

Deduplication and execution

We run the job with deduplication to prevent accidental duplicates:

def run(%Date{} = date \ Timex.shift(Timex.today(), days: -1), only_logs \ false) do
  %{day: day, month: month, year: year} = date

  job =
    %{date: %{day: day, month: month, year: year}, only_logs: only_logs}
    |> NextID.Job.MongodbDailyLog.new()

  with {:ok, %Oban.Job{conflict?: true}} <- Oban.insert(job) do
    {:error, :job_already_exists}
  end
end

Historical processing? No problem.

Need to backfill historical data? We created a method that recursively schedules jobs by date:

def run_history(%Date{} = date \ Timex.today()) do
  case Timex.before?(date, ~D[2021-08-01]) do
    false ->
      run(date, true)
      run_history(Timex.shift(date, days: -1))
    true -> :ok
  end
end

Results

We automated daily log processing with high reliability
Achieved horizontal scalability by splitting queues by task type
Eliminated duplication issues while maintaining traceability
Reduced manual work and reprocessing effort
Gained full visibility and control over all jobs — for now, straight from PostgreSQL

Conclusion

Oban turned out to be a robust, easy-to-implement, and well-integrated solution within the Elixir ecosystem. Today, it’s a foundational part of our billing system — and we’re already expanding its use to other parts of the product.

If you’re dealing with critical background processing like billing, I highly recommend giving it a try.

📚 Official Oban repo: github.com/oban-bg/oban

💡 About Nextcode: https://nxcd.com.br

💬 Want to chat about how we’re using it? Reach out!

How We Manage Billing Workflows in Elixir Using Oban