Monitoring Golang Services with Prometheus: Choosing Between Pull and Push Models

What is Prometheus and Why Do You Need It?

Prometheus is a powerful monitoring system that collects and processes numerical data (metrics) from applications. It helps you track key indicators such as:

• The number of requests handled by your service.
• The response time for each request.
• Memory and CPU usage.
• The number of errors occurring in the system.

By using Prometheus, you can answer critical questions like:

• "Is my service running efficiently?"

• "What are the performance bottlenecks?"

• "Do we need to scale up our infrastructure?"

And how does Prometheus Collect Metrics?

There are two primary ways Prometheus gathers data:

1. Pull model – Prometheus actively queries services for their metrics.
2. Push model (Pushgateway) – Services push their metrics to an intermediary, which Prometheus then collects.

Let’s break them down.

Pull Model

In the pull model, Prometheus actively fetches metrics from your application via HTTP (e.g., http://your-service:8080/metrics).

This is the default and most commonly used method.

Setting up Prometheus with Golang (Pull Model)

1. Install the necessary libraries:

   go get github.com/prometheus/client_golang/prometheus
   go get github.com/prometheus/client_golang/prometheus/promhttp
   

2. Define your metrics (e.g., counting HTTP requests):

   import (
       "github.com/prometheus/client_golang/prometheus"
       "github.com/prometheus/client_golang/prometheus/promauto"
   )
   
   var httpRequestsTotal = promauto.NewCounter(prometheus.CounterOpts{
       Name: "http_requests_total",
       Help: "Total number of HTTP requests",
   })
   

3. Expose a /metrics endpoint:

   import (
       "net/http"
   
       "github.com/prometheus/client_golang/prometheus/promhttp"
   )
   
   func main() {
       http.Handle("/metrics", promhttp.Handler())
   }
   

4. Configure Prometheus to scrape metrics from your service in prometheus.yml:

   scrape_configs:
     - job_name: "example_service"
       static_configs:
         - targets: ["localhost:8080"]
   

Now, Prometheus will automatically query http://localhost:8080/metrics every few seconds to collect data.

Why is the Pull Model Preferred?

• Simplicity – Prometheus controls the scraping schedule and frequency.
• Fewer points of failure – No need for an additional service to receive metrics.
• Automatic cleanup – If a service stops responding, Prometheus simply stops receiving data, avoiding stale metrics.

Push Model (Pushgateway Approach)

In the push model, a service sends its metrics to an intermediary service called Pushgateway, which stores them until Prometheus fetches them.

How it Works (Push Model)

1. Your application pushes metrics to Pushgateway:

   import (
       "github.com/prometheus/client_golang/prometheus"
       "github.com/prometheus/client_golang/prometheus/push"
   )
   
   func main() {
       registry := prometheus.NewRegistry()
       jobCounter := prometheus.NewCounter(prometheus.CounterOpts{
           Name: "job_execution_count",
           Help: "Number of executed jobs",
       })
   
       registry.MustRegister(jobCounter)
       jobCounter.Inc()
   
       err := push.New("http://localhost:9090", "my_service_or_job").
           Collector(jobCounter).
           Grouping("instance", "worker_1").
           Push()
       if err != nil {
           panic(err)
       }
   }
   

2. Configure Prometheus to collect data from Pushgateway:

   scrape_configs:
     - job_name: "pushgateway"
       static_configs:
         - targets: ["localhost:9091"]
   

When is the Push Model Actually Useful?

• Short-lived jobs (batch tasks, cron jobs) that complete before Prometheus can scrape them.
• Network restrictions where Prometheus cannot directly access the service.
• External data sources (IoT devices, external APIs) that cannot be scraped directly.

Which Model Should You Use?

Why Push Model is Not Ideal?

Although Pushgateway solves some problems (e.g., short-lived processes that terminate before Prometheus scrapes them), it introduces several new issues:

1. Difficult to manage stale data

   • If a service dies, its old metrics remain in Pushgateway.

   • Prometheus has no way of knowing if the service is still running.

   • You must manually delete outdated metrics using push.Delete(...) or configure expiry policies.

2. Additional complexity

   • Instead of a direct Service → Prometheus link, you now have Service → Pushgateway → Prometheus.

   • Pushgateway is an extra dependency, increasing maintenance overhead.

3. Potential bottlenecks

   • If many services push metrics frequently, Pushgateway can become overwhelmed.

   • Unlike direct Prometheus scrapes (which distribute the load), all requests hit a single Pushgateway instance.

4. Data consistency issues
   • If multiple services push metrics with the same name but different values, data may be overwritten, leading to incorrect results.

Conclusion

Prometheus is a powerful and reliable tool for monitoring services. For most applications, the pull model is the best choice—it's simple, efficient, and ensures fresh data without additional complexity. However, if you're working with short-lived processes like Lambda functions or batch jobs, the push model via Pushgateway can be useful to capture metrics before the process exits.

Choosing the right approach ensures better observability and maintainability of your system.

Take care!