add metric captureing to email service

Author: mollpo

pom.xml

@@ -49,6 +49,18 @@
             <artifactId>lombok</artifactId>
             <optional>true</optional>
         </dependency>
+        <!-- Optional: metrics are only recorded if the consuming application brings a MeterRegistry
+             (usually via spring-boot-starter-actuator). Without it the library falls back to a no-op. -->
+        <dependency>
+            <groupId>io.micrometer</groupId>
+            <artifactId>micrometer-core</artifactId>
+            <optional>true</optional>
+        </dependency>
+        <dependency>
+            <groupId>io.micrometer</groupId>
+            <artifactId>micrometer-registry-prometheus</artifactId>
+            <scope>test</scope>
+        </dependency>
         <dependency>
             <groupId>org.springframework.boot</groupId>
             <artifactId>spring-boot-starter-test</artifactId>

readme.md

@@ -56,6 +56,68 @@ To read email datasets from the database use this class: [QueryEmail.java](src%2
 
 If you want to receive a report after each run of the scheduler, create a Bean implementing [EmailSchedulerCallback.java](src%2Fmain%2Fjava%2Fit%2Faboutbits%2Fspringboot%2Femailservice%2Flib%2FEmailSchedulerCallback.java)
 
+### Metrics
+
+The library records Micrometer metrics for both schedulers. Micrometer is an optional dependency: if the
+application provides a `MeterRegistry` the metrics are recorded, otherwise the library falls back to a no-op
+and nothing changes. Nothing has to be enabled in this library.
+
+To scrape them, the application needs `spring-boot-starter-actuator` and `micrometer-registry-prometheus`,
+plus the Prometheus endpoint:
+
+```yaml
+management:
+  endpoints:
+    access:
+      default: none
+    web:
+      exposure:
+        include:
+          - health
+          - prometheus
+  endpoint:
+    health:
+      access: read-only
+    prometheus:
+      access: read-only
+```
+
+The following series are exposed:
+
+| Series                                          | Type    | Labels                                                                         | Description                                                                                     |
+|-------------------------------------------------|---------|--------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|
+| `app_email_send_duration_seconds`               | timer   | `mode`: `scheduled`, `direct`<br/>`outcome`: `sent`, `retry`, `error`           | One observation per send attempt. `retry` is an attempt that failed but is scheduled for another one, `error` an email that has given up. |
+| `app_email_cleanup_duration_seconds`            | timer   | `outcome`: `cleaned`, `error`                                                   | One observation per attachment cleanup attempt.                                                 |
+| `app_email_pass_duration_seconds`               | timer   | `job`: `send`, `cleanup`<br/>`status`: `success`, `failed`                      | One observation per scheduler pass. `failed` means the pass itself broke, e.g. the database was unreachable. |
+| `app_email_last_run_timestamp_seconds`          | gauge   | `job`: `send`, `cleanup`                                                        | When the scheduler last fired. Stalls if the scheduler is dead or the pod is down.               |
+| `app_email_last_success_timestamp_seconds`      | gauge   | `job`: `send`, `cleanup`                                                        | When a pass last got through. Stalls while passes keep failing.                                  |
+| `app_email_queue`                               | gauge   | `state`: `pending`, `sending`                                                   | Emails per state, read after each send pass. The terminal states `SENT` and `ERROR` are left out: they only ever grow. Errors are counted by `app_email_send_duration_seconds_count{outcome="error"}`. |
+| `app_email_queue_oldest_due_age_seconds`         | gauge   | `state`: `pending`                                                              | How long the oldest email that is already due has been waiting. `0` if nothing is due.           |
+
+Two things to keep in mind when querying them:
+
+- **Aggregate the gauges with `max`, never `sum`.** The queue gauges are read from the database, so every pod
+  reports the same numbers - summing them multiplies the backlog by the number of pods.
+- The queue gauges are only written by the send scheduler. With
+  `aboutbits.emailservice.scheduling.enabled=false` they are never registered, and the series are absent
+  rather than zero.
+
+Both timestamp gauges are registered on first use, so a pod that has not completed a pass since starting has
+no series at all. This is deliberate: an alert reads an absent series the same way it reads a `NaN` starting
+value, while a `0` starting value would look like decades of staleness after every deploy.
+
+The dashboards and alerts themselves belong to the consuming application, since they depend on how it is
+deployed. As a starting point, a scheduler that has stopped firing and a backlog that is not being kept up
+with read as:
+
+```promql
+time() - max by (job) (app_email_last_run_timestamp_seconds) > 300
+max(app_email_queue_oldest_due_age_seconds) > 600
+```
+
+If the application replaces the metrics sink with its own `EmailMetrics` bean, the library's one steps back;
+whatever the application provides is called fail-safe, so a failing sink can never break a send.
+
 ### Configuration
 
 To enable this service just add `@EnableEmailService` to your main class. You must also enable `@EnableScheduling` to allow the email queue to be processed.

src/main/java/it/aboutbits/springboot/emailservice/EmailServiceConfiguration.java

@@ -2,6 +2,7 @@
 
 import it.aboutbits.springboot.emailservice.lib.AttachmentCleanerCallback;
 import it.aboutbits.springboot.emailservice.lib.AttachmentDataSource;
+import it.aboutbits.springboot.emailservice.lib.EmailMetrics;
 import it.aboutbits.springboot.emailservice.lib.EmailSchedulerCallback;
 import it.aboutbits.springboot.emailservice.lib.application.CleanupAttachmentFiles;
 import it.aboutbits.springboot.emailservice.lib.application.EmailAttachmentMapper;
@@ -14,6 +15,7 @@
 import it.aboutbits.springboot.emailservice.lib.application.SendScheduledEmails;
 import it.aboutbits.springboot.emailservice.lib.application.UnavailableAttachmentDataSource;
 import it.aboutbits.springboot.emailservice.lib.jpa.EmailRepository;
+import it.aboutbits.springboot.emailservice.lib.metrics.FailSafeEmailMetrics;
 import org.jspecify.annotations.NullMarked;
 import org.springframework.beans.factory.annotation.Value;
 import org.springframework.boot.autoconfigure.AutoConfigurationPackage;
@@ -27,6 +29,10 @@
 import java.time.Duration;
 import java.util.List;
 
+/*
+ * Imported through EmailServiceImportSelector, never directly: see there for why the import is deferred
+ * and where the EmailMetrics bean comes from.
+ */
 @AutoConfigurationPackage
 @NullMarked
 public class EmailServiceConfiguration {
@@ -52,11 +58,13 @@ public QueryEmail queryEmail(EmailRepository emailRepository, EmailMapper emailM
     }
 
     @Bean
+    @SuppressWarnings("checkstyle:ParameterNumber")
     public ManageEmail manageEmail(
             EmailRepository emailRepository,
             JavaMailSender javaMailSender,
             AttachmentDataSource attachmentDataSource,
             EmailMapper emailMapper,
+            EmailMetrics emailMetrics,
             @Value("${aboutbits.emailservice.scheduling.max-attempts:3}") int maxAttempts,
             @Value("${aboutbits.emailservice.scheduling.interval:30000}") long schedulerIntervalMillis,
             PlatformTransactionManager transactionManager
@@ -66,6 +74,7 @@ public ManageEmail manageEmail(
                 javaMailSender,
                 attachmentDataSource,
                 emailMapper,
+                failSafe(emailMetrics),
                 maxAttempts,
                 Duration.ofMillis(schedulerIntervalMillis),
                 transactionManager
@@ -78,9 +87,16 @@ public SendScheduledEmails sendScheduledEmails(
             QueryEmail queryEmail,
             ManageEmail manageEmail,
             List<EmailSchedulerCallback> callbacks,
+            EmailMetrics emailMetrics,
             @Value("${aboutbits.emailservice.scheduling.stuck-sending-recovery-threshold:PT30M}") Duration stuckSendingRecoveryThreshold
     ) {
-        return new SendScheduledEmails(queryEmail, manageEmail, callbacks, stuckSendingRecoveryThreshold);
+        return new SendScheduledEmails(
+                queryEmail,
+                manageEmail,
+                callbacks,
+                failSafe(emailMetrics),
+                stuckSendingRecoveryThreshold
+        );
     }
 
     @Bean
@@ -89,14 +105,28 @@ public CleanupAttachmentFiles cleanupAttachments(
             QueryEmail queryEmail,
             ManageEmail manageEmail,
             List<AttachmentCleanerCallback> callbacks,
+            EmailMetrics emailMetrics,
             @Value("${aboutbits.emailservice.scheduling.stuck-cleanup-recovery-threshold:PT30M}") Duration stuckCleanupRecoveryThreshold
     ) {
-        return new CleanupAttachmentFiles(queryEmail, manageEmail, callbacks, stuckCleanupRecoveryThreshold);
+        return new CleanupAttachmentFiles(
+                queryEmail,
+                manageEmail,
+                callbacks,
+                failSafe(emailMetrics),
+                stuckCleanupRecoveryThreshold
+        );
     }
 
     @Bean
     @ConditionalOnMissingBean(AttachmentDataSource.class)
     public AttachmentDataSource attachmentDataSource() {
         return new UnavailableAttachmentDataSource();
     }
+
+    // The EmailMetrics bean itself stays whatever the application sees - the library's own or a
+    // replacement - but nothing inside the library talks to it unguarded: an email that went out over
+    // SMTP has to be persisted as SENT no matter what a metrics backend does.
+    private static EmailMetrics failSafe(EmailMetrics emailMetrics) {
+        return new FailSafeEmailMetrics(emailMetrics);
+    }
 }

src/main/java/it/aboutbits/springboot/emailservice/EmailServiceImportSelector.java

@@ -0,0 +1,38 @@
+package it.aboutbits.springboot.emailservice;
+
+import org.jspecify.annotations.NullMarked;
+import org.springframework.context.annotation.DeferredImportSelector;
+import org.springframework.core.Ordered;
+import org.springframework.core.type.AnnotationMetadata;
+
+/*
+ * Registers the library the way an auto-configuration is registered: only after every configuration
+ * class of the application has been parsed. That is what lets the @ConditionalOnMissingBean fallbacks
+ * (AttachmentDataSource, EmailMetrics) see a replacement the application declares, even one declared
+ * in the very class that carries @EnableEmailService - a plain @Import is processed before the
+ * importing class, so such a replacement would only be found after the fallback has already been
+ * registered.
+ *
+ * The metrics configurations are listed here instead of being nested in EmailServiceConfiguration:
+ * Spring only picks up nested configuration classes below a class annotated @Configuration, and
+ * EmailServiceConfiguration is not one.
+ */
+@NullMarked
+public class EmailServiceImportSelector implements DeferredImportSelector, Ordered {
+    @Override
+    public String[] selectImports(AnnotationMetadata importingClassMetadata) {
+        return new String[]{
+                EmailServiceConfiguration.class.getName(),
+                MicrometerEmailMetricsConfiguration.class.getName(),
+                NoOpEmailMetricsConfiguration.class.getName()
+        };
+    }
+
+    // Spring Boot's auto-configuration runs at LOWEST_PRECEDENCE - 1 and expects the
+    // @AutoConfigurationPackage of EmailServiceConfiguration to be registered by then, since that is
+    // where it finds the entities and repositories of this library.
+    @Override
+    public int getOrder() {
+        return Ordered.LOWEST_PRECEDENCE - 2;
+    }
+}

src/main/java/it/aboutbits/springboot/emailservice/EnableEmailService.java

@@ -9,6 +9,6 @@
 
 @Target({ElementType.TYPE})
 @Retention(RetentionPolicy.RUNTIME)
-@Import({EmailServiceConfiguration.class})
+@Import({EmailServiceImportSelector.class})
 public @interface EnableEmailService {
 }

src/main/java/it/aboutbits/springboot/emailservice/MicrometerEmailMetricsConfiguration.java

@@ -0,0 +1,38 @@
+package it.aboutbits.springboot.emailservice;
+
+import io.micrometer.core.instrument.MeterRegistry;
+import it.aboutbits.springboot.emailservice.lib.EmailMetrics;
+import it.aboutbits.springboot.emailservice.lib.metrics.MicrometerEmailMetrics;
+import it.aboutbits.springboot.emailservice.lib.metrics.NoOpEmailMetrics;
+import org.jspecify.annotations.NullMarked;
+import org.springframework.beans.factory.ObjectProvider;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+
+/*
+ * Micrometer is an optional dependency, so this configuration and NoOpEmailMetricsConfiguration are
+ * guarded by mutually exclusive class conditions rather than by bean ordering: exactly one of them ever
+ * applies, and this one, the only one referencing MeterRegistry, is only loaded once that class exists.
+ */
+@Configuration(proxyBeanMethods = false)
+@ConditionalOnClass(name = "io.micrometer.core.instrument.MeterRegistry")
+@NullMarked
+class MicrometerEmailMetricsConfiguration {
+    // Micrometer on the classpath without a registry bean - micrometer-core without actuator - still
+    // leaves nothing to record into, hence the fallback rather than a required dependency. The same goes
+    // for several registries without a primary one: a library that only records "if a registry happens
+    // to be there" must not fail the whole context over an ambiguity it cannot resolve.
+    @Bean
+    @ConditionalOnMissingBean(EmailMetrics.class)
+    EmailMetrics emailMetrics(ObjectProvider<MeterRegistry> meterRegistry) {
+        var registry = meterRegistry.getIfUnique();
+
+        if (registry == null) {
+            return new NoOpEmailMetrics();
+        }
+
+        return new MicrometerEmailMetrics(registry);
+    }
+}

src/main/java/it/aboutbits/springboot/emailservice/NoOpEmailMetricsConfiguration.java

@@ -0,0 +1,21 @@
+package it.aboutbits.springboot.emailservice;
+
+import it.aboutbits.springboot.emailservice.lib.EmailMetrics;
+import it.aboutbits.springboot.emailservice.lib.metrics.NoOpEmailMetrics;
+import org.jspecify.annotations.NullMarked;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingClass;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+
+// Counterpart of MicrometerEmailMetricsConfiguration for applications without Micrometer on the classpath.
+@Configuration(proxyBeanMethods = false)
+@ConditionalOnMissingClass("io.micrometer.core.instrument.MeterRegistry")
+@NullMarked
+class NoOpEmailMetricsConfiguration {
+    @Bean
+    @ConditionalOnMissingBean(EmailMetrics.class)
+    EmailMetrics emailMetrics() {
+        return new NoOpEmailMetrics();
+    }
+}

src/main/java/it/aboutbits/springboot/emailservice/lib/EmailMetrics.java

@@ -0,0 +1,86 @@
+package it.aboutbits.springboot.emailservice.lib;
+
+import org.jspecify.annotations.NullMarked;
+import org.jspecify.annotations.Nullable;
+
+import java.time.Duration;
+import java.util.function.Supplier;
+
+/*
+ * Sink for everything the schedulers learn while running. Implemented by the library against
+ * Micrometer when the consuming application provides a MeterRegistry, and by a no-op otherwise,
+ * so that neither the schedulers nor their callers need to know whether metrics are enabled.
+ */
+@NullMarked
+public interface EmailMetrics {
+    void sendAttempt(SendMode mode, SendOutcome outcome, Duration duration);
+
+    void cleanupAttempt(CleanupOutcome outcome, Duration duration);
+
+    void pass(Job job, PassStatus status, Duration duration);
+
+    // Takes a supplier because the snapshot costs database queries: a sink that has no backend to
+    // record into is expected to never read it.
+    void queue(Supplier<QueueSnapshot> snapshot);
+
+    /*
+     * Runs a whole scheduler pass and records it, so that a database outage - the one failure the pass
+     * itself cannot report per email - still shows up as a failed pass before it propagates to the
+     * scheduler. Recorded in a finally block on purpose: an Error is a fired pass as much as an
+     * exception is, and last_run is defined as the time the scheduler last fired.
+     */
+    default void timedPass(Job job, Runnable pass) {
+        var startNanos = System.nanoTime();
+        var status = PassStatus.FAILED;
+
+        try {
+            pass.run();
+            status = PassStatus.SUCCESS;
+        } finally {
+            pass(job, status, Duration.ofNanos(System.nanoTime() - startNanos));
+        }
+    }
+
+    enum SendMode {
+        SCHEDULED,
+        DIRECT
+    }
+
+    enum SendOutcome {
+        SENT,
+        // The attempt failed but the email is scheduled for another one, so it is not an error yet.
+        RETRY,
+        ERROR
+    }
+
+    enum CleanupOutcome {
+        CLEANED,
+        ERROR
+    }
+
+    enum Job {
+        SEND,
+        CLEANUP
+    }
+
+    enum PassStatus {
+        SUCCESS,
+        FAILED
+    }
+
+    /*
+     * State of the queue as a whole, read once per scheduler pass. Since every pod reads the same
+     * rows, the resulting series are identical across pods and must be aggregated with max(), not sum().
+     *
+     * Only the two states an email passes through are counted. SENT and ERROR are terminal, so their
+     * counts only ever grow and say nothing about how the queue is doing right now; how many emails
+     * end in an error is already answered by the outcome tag of the send attempts.
+     */
+    record QueueSnapshot(
+            long pending,
+            long sending,
+            // How long the oldest email that is already due has been waiting; null if nothing is due.
+            @Nullable Duration oldestDueAge
+    ) {
+    }
+}