Welcome To Fusebes - Dev & Programming Blog

Build A Web Crawler with Java
02
Jan
2022

Build A Web Crawler with Java

Creating a web crawler is a smart way of retrieving useful information available online. With a web crawler, you can scan the Internet, browse through individual websites, and analyze and extract their content.

The Java programming language provides a simple way of building a web crawler and harvesting data from websites. You can use the extracted data for various use cases, such as for analytical purposes, providing a service that uses third-party data, or generating statistical data.

In this article, we’ll walk you through the process of building a web crawler using Java and ProxyCrawl.

What you’ll need

Typically, crawling web data involves creating a script that sends a request to the targeted web page, accesses its underlying HTML code, and scrapes the required information.

To accomplish that objective, you’ll need the following:

Before we develop the crawling logic, let’s clear the air why using ProxyCrawl is important for web crawling.

Why use ProxyCrawl for Crawling

ProxyCrawl is a powerful data crawling and scraping tool you can use to harvest information from websites fast and easily.

Here are some reasons why you should use it for crawling online data:

  • Easy to use It comes with a simple API that you can set up quickly without any programming hurdles. With just a few lines of code, you can start using the API to crawl websites and retrieve their content.
  • Supports advanced crawling ProxyCrawl allows you to perform advanced web crawling and scrape data from complicated websites. Since it supports JavaScript rendering, ProxyCrawl lets you extract data from dynamic websites. It offers a headless browser that allows you to extract what real users see on their web browsers—even if a site is created using modern frameworks like Angular or React.js.
  • Bypass crawling obstacles ProxyCrawl can handle all the restrictions often associated with crawling online data. It has an extensive network of proxies as well as more than 17 data centers around the world. You can use it to avoid access restrictions, resolve CAPTCHAs, and evade other anti-scraping measures implemented by web applications. What’s more, you can crawl websites while remaining anonymous; you’ll not worry about exposing your identity.
  • Free trial account You can test how ProxyCrawl works without giving out your payment details. The free account comes with 1,000 credits for trying out the tool’s capabilities.

How ProxyCrawl Works

ProxyCrawl provides the Crawling API for crawling and scraping data from websites. You can easily integrate the API in your Java development project and retrieve information from web pages smoothly.

Each request made to the Crawling API starts with the following base part:

https://api.proxycrawl.com

Also, you’ll need to add the following mandatory parameters to the API:

  • Authentication token
  • URL

The authentication token is a unique token that authorizes you to use the Crawling API. Once you sign up for an account, ProxyCrawl will give you two types of tokens:

  • Normal token This is for making generic crawling requests.
  • JavaScript token This is for crawling dynamic websites. It provides you with headless browser capabilities for crawling web pages rendered using JavaScript. As pointed out earlier, it’s a useful way of crawling advanced websites.

Here is how to add the authentication token to your API request:

https://api.proxycrawl.com/?token=INSERT_TOKEN

The second mandatory parameter is the URL to crawl. It should start with HTTP or HTTPS, and be completely encoded. Encoding converts the URL string into a format that can be transferred over the Internet validly and easily.

Here is how to insert the URL to your API request:

https://api.proxycrawl.com/?token=INSERT_TOKEN&url=INSERT_URL

If you run the above line—for example, on your terminal using cURL or pasting it on a browser’s address bar—it’ll execute the API request and return the entire HTML source code of the targeted web page.

It’s that easy and simple!

If you want to perform advanced crawling, you may add other parameters to the API request. For example, when using the JavaScript token, you can add the page_wait parameter to instruct the browser to wait for the specified number of milliseconds before the resulting HTML code is captured.

Here is an example:

https://api.proxycrawl.com/?token=INSERT_TOKEN&page_wait=1000&url=INSERT_URL

Building a Web Crawler in Java and ProxyCrawl

In this Java web crawling tutorial, we’ll use the HttpClient API to create the crawling logic. The API was introduced in Java 11, and it comes with lots of useful features for sending requests and retrieving their responses.

The HttpClient API supports both HTTP/1.1 and HTTP/2. By default, it uses the HTTP/2 protocol to send requests. If a request is sent to a server that does not already support HTTP/2, it will automatically be downgraded to HTTP/1.

Furthermore, its requests can be sent asynchronously or synchronously, it handles requests and response bodies as reactive-streams, and uses the common builder pattern.

The API is comprised of three core classes:

  • HttpRequest
  • HttpClient
  • HttpResponse

Let’s talk about each of them in more detail.

1. HttpRequest

The HttpRequest, as the name implies, is an object encapsulating the HTTP request to be sent. To create new instances of HttpRequest, call HttpRequest.newBuilder(). After it has been created, the request is immutable and can be sent multiple times.

The Builder class comes with different methods for configuring the request.

These are the most common methods:

  • URI method
  • Request method
  • Protocol version method
  • Timeout method

Let’s talk about each of them in more detail.

a) URI method

The first thing to do when configuring the request is to set the URL to crawl. We can do so by calling the uri() method on the Builder instance. We’ll also use the URI.create() method to create the URI by parsing the string of the URL we intend to crawl.

Here is the code:

String url =
URLEncoder.encode(“https://www.forextradingbig.com/7-reasons-why-you-should
-quit-forex-trading/”, StandardCharsets.UTF_8.name());

HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(“https://api.proxycrawl.com/?token=INSERT_TOKEN&url=”
+ url))

Notice that we provided the URL string using ProxyCrawl’s settings. This is the web page we intend to scrape its contents.

We also encoded the URL using the Java URLEncoder class. As earlier mentioned, ProxyCrawl requires URLs to be encoded.

b) Request method

The next thing to do is to specify the HTTP method to be used for making the request. We can call any of the following methods from Builder:

  • GET()
  • POST()
  • PUT()
  • DELETE()

In this case, since we want to request data from the target web page, we’ll use the GET() method.

Here is the code:

HttpRequest request = HttpRequest.newBuilder()
.GET()

So far, HttpRequest has all the parameters that should be passed to HttpClient. However, you may need to include other parameters, such as the HTTP protocol version and timeout.

Let’s see how you can add the additional parameters.

c) Protocol version method

As earlier mentioned, the HttpClient API uses the HTTP/2 protocol by default. Nonetheless, you can specify the version of the HTTP protocol you want to use.

Here is the code:

HttpRequest request = HttpRequest.newBuilder()
.version(HttpClient.Version.HTTP_2)

d) Timeout method

You can set the amount of time to wait before a response is received. Once the defined period expires, an HttpTimeoutException will be thrown. By default, the timeout is set to infinity.

You can define timeout by calling the timeout() method on the builder instance. You’ll also need to pass the Duration object to specify the amount of time to wait.

Here is the code:

HttpRequest request = HttpRequest.newBuilder()
.timeout(Duration.ofSeconds(20))

2. HttpClient

The HttpClient class is the main entry point of the API—it acts as a container for the configuration details shared among multiple requests. It is the HTTP client used for sending requests and receiving responses.

You can call either the HttpClient.newBuilder() or the HttpClient.newHttpClient() method to instantiate it. After an instance of the HttpClient has been created, it’s immutable.

The HttpClient class offers several helpful and self-describing methods you can use when working with requests and responses.

These are some things you can do:

  • Set protocol version
  • Set redirect policy
  • Send synchronous and asynchronous requests

Let’s talk about each of them in more detail.

a) Set protocol version

As earlier mentioned, the HttpClient class uses the HTTP/2 protocol by default. However, you can set your preferred protocol version, either HTTP/1.1 or HTTP/2.

Here is an example:

HttpClient client = HttpClient.newBuilder()
.version(Version.HTTP_1_1)

b) Set redirect policy

If the targeted web page has moved to a different address, you’ll get a 3xx HTTP status code. Since the address of the new URI is usually provided with the status code information, setting the correct redirect policy can make HttpClient forward the request automatically to the new location.

You can set it by using the followRedirects() method on the Builder instance.

Here is an example:

HttpClient client = HttpClient.newBuilder()
.followRedirects(Redirect.NORMAL)

c) Send synchronous and asynchronous requests

HttpClient supports two ways of sending requests:

  • Synchronously by using the send() method. This blocks the client until the response is received, before continuing with the rest of the execution.

Here is an example:

HttpResponse<String> response = client.send(request,
BodyHandlers.ofString());

Note that we used BodyHandlers and called the ofString() method to return the HTML response as a string.

  • Asynchronously by using the sendAsync() method. This does not wait for the response to be received; it’s non-blocking. Once the sendAsync() method is called, it returns instantly with a CompletableFuture< HttpResponse >, which finalizes once the response is received. The returned CompletableFuture can be joined using various techniques to define dependencies among varied asynchronous tasks.

Here is an example:

CompletableFuture<HttpResponse<String>> response = HttpClient.newBuilder()
.sendAsync(request, HttpResponse.BodyHandler.ofString());

3. HttpResponse

The HttpResponse, as the name implies, represents the response received after sending an HttpRequest. HttpResponse offers different helpful methods for handling the received response.

These are the most important methods:

  • statusCode() This method returns the status code of the response. It’s of int type
  • Body() This method returns a body for the response. The return type is based on the kind of response BodyHandler parameter that is passed to the send() method.

Here is an example:

// Handling the response body as a String
HttpResponse<String> response = client
.send(request, BodyHandlers.ofString());

// Printing response body
System.out.println(response.body());

// Printing status code
System.out.println(response.statusCode());
// Handling the response body as a file
HttpResponse<Path> response = client
.send(request, BodyHandlers.ofFile(Paths.get(“myexample.html”)));

Synchronous Example

Here is an example that uses the HttpClient synchronous method to crawl a web page and output its content:

package javaHttpClient;

import java.io.IOException;
import java.net.URI;
import java.net.URLEncoder;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
import java.nio.charset.StandardCharsets;

public class SyncExample {

public static void main(String[] args) throws IOException, InterruptedException {

// Encoding the URL
String url = URLEncoder.encode(“https://www.forextradingbig.com/7-reasons-why-you-should-quit-forex-trading/”, StandardCharsets.UTF_8.name());

// Instantiating HttpClient
HttpClient client = HttpClient.newHttpClient();

// Configuring HttpRequest
HttpRequest request = HttpRequest.newBuilder()
.GET()
.uri(URI.create(“https://api.proxycrawl.com/?token=INSERT_TOKEN&url=” + url))
.build();

// Handling the response
HttpResponse<String> response = client.send(request, BodyHandlers.ofString());
System.out.println(response.body());
}

}

Here is the output:

Asynchronous Example

When using the HttpClient asynchronous method to crawl a web page, the sendAsync() method is called, instead of send().

Here is an example:

package javaHttpClient;

import java.io.IOException;
import java.net.URI;
import java.net.URLEncoder;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;

public class AsyncExample {


public static void main(String[] args) throws IOException, InterruptedException, ExecutionException, TimeoutException {

// Encoding the URL
String url = URLEncoder.encode(“https://www.forextradingbig.com/7-reasons-why-you-should-quit-forex-trading/”, StandardCharsets.UTF_8.name());

// Instantiating HttpClient
HttpClient client = HttpClient.newHttpClient();

// Configuring HttpRequest
HttpRequest request = HttpRequest.newBuilder()
.GET()
.version(HttpClient.Version.HTTP_2)
.uri(URI.create(“https://api.proxycrawl.com/?token=INSERT_TOKEN&url=” + url))
.build();

// Handling the response
CompletableFuture<HttpResponse<String>> response =
client.sendAsync(request, HttpResponse.BodyHandlers.ofString());

String result = response.thenApply(HttpResponse::body).get(5, TimeUnit.SECONDS);

System.out.println(result);

}

}

Conclusion

That’s how to build a web crawler in Java. The HttpClient API, which was introduced in Java 11, makes it easy to send and handle responses from a server.

And if the API is combined with a versatile tool like ProxyCrawl, it can make web crawling tasks smooth and rewarding.

With ProxyCrawl, you can create a scraper that can help you to retrieve information from websites anonymously and without worrying about being blocked.

It’s the tool you need to take your crawling efforts to the next level.

Click here to create a free ProxyCrawl account.

Happy scraping!

Spring Boot Actuator metrics monitoring with Prometheus and Grafana
03
Mar
2021

Spring Boot Actuator metrics monitoring with Prometheus and Grafana

Welcome to the second part of the Spring Boot Actuator tutorial series. In the first part, you learned what spring-boot-actuator module does, how to configure it in a spring boot application, and how to interact with various actuator endpoints.

In this article, you’ll learn how to integrate spring boot actuator with a monitoring system called Prometheus and a graphing solution called Grafana.

At the end of this article, you’ll have a Prometheus as well as a Grafana dashboard setup in your local machine where you’ll be able to visualize and monitor all the metrics generated from the Spring Boot application.

Prometheus

Prometheus is an open-source monitoring system that was originally built by SoundCloud. It consists of the following core components –

  • A data scraper that pulls metrics data over HTTP periodically at a configured interval.
  • time-series database to store all the metrics data.
  • A simple user interface where you can visualize, query, and monitor all the metrics.

Grafana

Grafana allows you to bring data from various data sources like Elasticsearch, Prometheus, Graphite, InfluxDB etc, and visualize them with beautiful graphs.

It also lets you set alert rules based on your metrics data. When an alert changes state, it can notify you over email, slack, or various other channels.

Note that, Prometheus dashboard also has simple graphs. But Grafana’s graphs are way better. That’s why, in this post, we’ll integrate Grafana with Prometheus to import and visualize our metrics data.

Adding Micrometer Prometheus Registry to your Spring Boot application

Spring Boot uses Micrometer, an application metrics facade to integrate actuator metrics with external monitoring systems.

It supports several monitoring systems like Netflix Atlas, AWS Cloudwatch, Datadog, InfluxData, SignalFx, Graphite, Wavefront, Prometheus etc.

To integrate actuator with Prometheus, you need to add the micrometer-registry-prometheus dependency –

<!-- Micrometer Prometheus registry  -->
<dependency>
	<groupId>io.micrometer</groupId>
	<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Once you add the above dependency, Spring Boot will automatically configure a PrometheusMeterRegistry and a CollectorRegistry to collect and export metrics data in a format that can be scraped by a Prometheus server.

All the application metrics data are made available at an actuator endpoint called /prometheus. The Prometheus server can scrape this endpoint to get metrics data periodically.

Exploring Spring Boot Actuator’s /prometheus Endpoint

Let’s explore the prometheus endpoint that is exposed by Spring Boot when micrometer-registry-prometheus dependency is available on the classpath.

First of all, you’ll start seeing the prometheus endpoint on the actuator endpoint-discovery page (http://localhost:8080/actuator) –

Spring Boot Actuator Prometheus Endpoint

The prometheus endpoint exposes metrics data in a format that can be scraped by a Prometheus server. You can see the exposed metrics data by navigating to the prometheus endpoint (http://localhost:8080/actuator/prometheus) –

# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{id="direct",} 81920.0
jvm_buffer_memory_used_bytes{id="mapped",} 0.0
# HELP jvm_threads_live The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live gauge
jvm_threads_live 23.0
# HELP tomcat_global_received_bytes_total  
# TYPE tomcat_global_received_bytes_total counter
tomcat_global_received_bytes_total{name="http-nio-8080",} 0.0
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",cause="Allocation Failure",} 7.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="Allocation Failure",} 0.232
jvm_gc_pause_seconds_count{action="end of minor GC",cause="Metadata GC Threshold",} 1.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="Metadata GC Threshold",} 0.01
jvm_gc_pause_seconds_count{action="end of major GC",cause="Metadata GC Threshold",} 1.0
jvm_gc_pause_seconds_sum{action="end of major GC",cause="Metadata GC Threshold",} 0.302
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
# TYPE jvm_gc_pause_seconds_max gauge
jvm_gc_pause_seconds_max{action="end of minor GC",cause="Allocation Failure",} 0.0
jvm_gc_pause_seconds_max{action="end of minor GC",cause="Metadata GC Threshold",} 0.0
jvm_gc_pause_seconds_max{action="end of major GC",cause="Metadata GC Threshold",} 0.0
# HELP jvm_gc_live_data_size_bytes Size of old generation memory pool after a full GC
# TYPE jvm_gc_live_data_size_bytes gauge
jvm_gc_live_data_size_bytes 5.0657472E7

## More data ...... (Omitted for brevity)

Downloading and Running Prometheus using Docker

1. Downloading Prometheus

You can download the Prometheus docker image using docker pull command like so –

$ docker pull prom/prometheus

Once the image is downloaded, you can type docker image ls command to view the list of images present locally –

$ docker image ls
REPOSITORY                                   TAG                 IMAGE ID            CREATED             SIZE
prom/prometheus                              latest              b82ef1f3aa07        5 days ago          119MB

2. Prometheus Configuration (prometheus.yml)

Next, We need to configure Prometheus to scrape metrics data from Spring Boot Actuator’s /prometheus endpoint.

Create a new file called prometheus.yml with the following configurations –

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'spring-actuator'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    static_configs:
    - targets: ['HOST_IP:8080']

The above configuration file is an extension of the basic configuration file available in the Prometheus documentation.

The most important stuff to note in the above configuration file is the spring-actuator job inside scrape_configs section.

The metrics_path is the path of the Actuator’s prometheus endpoint. The targets section contains the HOST and PORT of your Spring Boot application.

Please make sure to replace the HOST_IP with the IP address of the machine where your Spring Boot application is running. Note that, localhost won’t work here because we’ll be connecting to the HOST machine from the docker container. You must specify the network IP address.

3. Running Prometheus using Docker

Finally, Let’s run Prometheus using Docker. Type the following command to start a Prometheus server in the background –

$ docker run -d --name=prometheus -p 9090:9090 -v <PATH_TO_prometheus.yml_FILE>:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml

Please make sure to replace the <PATH_TO_prometheus.yml_FILE> with the PATH where you have stored the Prometheus configuration file.

After running the above command, docker will start the Prometheus server inside a container. You can see the list of all the containers with the following command –

$ docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
e036eb20b8ad        prom/prometheus     "/bin/prometheus --c…"   4 minutes ago       Up 4 minutes        0.0.0.0:9090->9090/tcp   prometheus

4. Visualizing Spring Boot Metrics from Prometheus dashboard

That’s it! You can now navigate to http://localhost:9090 to explore the Prometheus dashboard.

You can enter a Prometheus query expression inside the Expression text field and visualize all the metrics for that query.

Following are some Prometheus graphs for our Spring Boot application’s metrics –

  • System’s CPU usage –
Spring Boot Actuator Metrics Dashboard Prometheus
  • Response latency of a slow API –
Spring Boot Actuator Prometheus Dashboard Graph Example

You can check out the official Prometheus documentation to learn more about Prometheus Query Expressions.

Downloading and running Grafana using Docker

Type the following command to download and run Grafana using Docker –

$ docker run -d --name=grafana -p 3000:3000 grafana/grafana 

The above command will start Grafana inside a Docker container and make it available on port 3000 on the Host machine.

You can type docker container ls to see the list of Docker containers –

$ docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED                  STATUS              PORTS                    NAMES
cf9196b30d0d        grafana/grafana     "/run.sh"                Less than a second ago   Up 5 seconds        0.0.0.0:3000->3000/tcp   grafana
e036eb20b8ad        prom/prometheus     "/bin/prometheus --c…"   16 minutes ago           Up 16 minutes       0.0.0.0:9090->9090/tcp   prometheus

That’s it! You can now navigate to http://localhost:3000 and log in to Grafana with the default username admin and password admin.

Configuring Grafana to import metrics data from Prometheus

Follow the steps below to import metrics from Prometheus and visualize them on Grafana:

1. Add the Prometheus data source in Grafana

Spring Boot Actuator Prometheus Grafana Dashboard

2. Create a new Dashboard with a Graph

Spring Boot Actuator Grafana Dashboard Create Graph

3. Add a Prometheus Query expression in Grafana’s query editor

Spring Boot Actuator Grafana Dashboard Prometheus Metrics Graph

4. Visualize metrics from Grafana’s dashboard

Spring Boot Actuator Grafana Dashboard Visualize Prometheus metrics graph

Read the First Part: Spring Boot Actuator: Health check, Auditing, Metrics gathering and Monitoring

More Learning Resources

Log4J Asynchronous Logging
03
Mar
2021

Log4J Asynchronous Logging

Asynchronous Logging

Log4j2 Supports Async loggers. These loggers provide a drastic improvement in performance compared to their synchronous counterparts.

Async Loggers internally use a library called Disruptor for asynchronous logging.

We need to include Disruptor dependency for using async loggers. Add the following to your pom.xml file –

<!-- Needed for Async Logging with Log4j 2 -->
<dependency>
    <groupId>com.lmax</groupId>
    <artifactId>disruptor</artifactId>
    <version>3.3.6</version>
</dependency>
Kotlin Infix Notation - Make function calls more intuitive
07
Mar
2021

Kotlin Classes, Objects, Constructors and Initializers

Kotlin supports method calls of a special kind, called infix calls.

You can mark any member function or extension function with the infix modifier to allow it to be called using infix notation. The only requirement is that the function should have only one required parameter.

Infix notations are used extensively in Kotlin. If you’ve been programming in Kotlin, chances are that you’ve already used infix notations.

Following are few common examples of infix notations in Kotlin –

1. Infix Notation Example – Creating a Map

val map = mapOf(1 to "one", 2 to "two", 3 to "three")

In the above example, the expressions 1 to "one"2 to "two" etc, are infix notations of the function calls 1.to("one") and 2.to("two") etc.

to() is an infix function that creates a Pair<A, B> from two values.

2. Infix Notation Example – Range Operators (until, downTo, step)

Kotlin provides various range operators that are usually called using infix notation –

for(i in 1 until 10) {	// Same as - for(i in 1.until(10))
    print("$i ")
}
for(i in 10 downTo 1) {	 // Same as - for(i in 10.downTo(1))
    print("$i ")
}
for(i in 1 until 10 step 2) { // Same as - for(i in 1.until(10).step(2))
    print("$i ")
}

3. Infix Notation Example – String.matches()

The String.matches() function in Kotlin which matches a String with a Regex is an infix function –

val regex = Regex("[tT]rue|[yY]es")
val str = "yes"

str.matches(regex)

// Infix notation of the above function call -
str matches regex

Creating an Infix Function

You can make a single argument member function or extension function, an infix function by marking it with the infix keyword.

Check out the following example where I have created an infix member function called add() for adding two Complex numbers –

data class ComplexNumber(val realPart: Double, val imaginaryPart: Double) {
	// Infix function for adding two complex numbers
    infix fun add(c: ComplexNumber): ComplexNumber {
        return ComplexNumber(realPart + c.realPart, imaginaryPart + c.imaginaryPart)
    }
}

You can now call the add() method using infix notation –

val c1 = ComplexNumber(3.0, 5.0)
val c2 = ComplexNumber(4.0, 7.0)

// Usual call
c1.add(c2) // produces - ComplexNumber(realPart=7.0, imaginaryPart=12.0)

// Infix call
c1 add c2  // produces - ComplexNumber(realPart=7.0, imaginaryPart=12.0)

Conclusion

That’s all folks. In this article, You learned what infix notation is and how it works. You saw several examples of Infix notations in Kotlin and also learned how to create an infix function.

Three Number Sum Solution
06
Feb
2021

Three Number Sum Solution

Three Number Sum Problem Statement

Given an array of integers, find all triplets in the array that sum up to a given target value.

In other words, given an array arr and a target value target, return all triplets a, b, c such that a + b + c = target.

Example:

Input array: [7, 12, 3, 1, 2, -6, 5, -8, 6]
Target sum: 0

Output: [[2, -8, 6], [3, 5, -8], [1, -6, 5]]

Three Number Sum Problem solution in Java

METHOD 1. Naive approach: Use three for loops

The naive approach is to just use three nested for loops and check if the sum of any three elements in the array is equal to the given target.

Time complexity: O(n^3)

import java.util.Scanner;
import java.util.List;
import java.util.ArrayList;
import java.util.Arrays;

class ThreeSum {

  // Time complexity: O(n^3)
  private static List<Integer[]> findThreeSum_BruteForce(int[] nums, int target) {
    List<Integer[]> result = new ArrayList<>();
    for (int i = 0; i < nums.length; i++) {
      for (int j = i + 1; j < nums.length; j++) {
        for (int k = j + 1; k < nums.length; k++) {
          if (nums[i] + nums[j] + nums[k] == target) {
            result.add(new Integer[] { nums[i], nums[j], nums[k] });
          }
        }
      }
    }
    return result;
  }

  public static void main(String[] args) {
    Scanner keyboard = new Scanner(System.in);

    int n = keyboard.nextInt();
    int[] nums = new int[n];

    for (int i = 0; i < n; i++) {
      nums[i] = keyboard.nextInt();
    }
    int target = keyboard.nextInt();

    keyboard.close();

    List<Integer[]> result = findThreeSum_Sorting(nums, target);

    for(Integer[] triplets: result) {
      for(int num: triplets) {
        System.out.print(num + " ");
      }
      System.out.println();
    }
  }
}

METHOD 2. Use Sorting along with the two-pointer sliding window approach

Another approach is to first sort the array, then –

  • Iterate through each element of the array and for every iteration,
    • Fix the first element (nums[i])
    • Try to find the other two elements whose sum along with nums[i] gives target. This boils down to the two sum problem.

Time complexity: O(n^2)

import java.util.Scanner;
import java.util.List;
import java.util.ArrayList;
import java.util.Arrays;

class ThreeSum {

  // Time complexity: O(n^2)
  private static List<Integer[]> findThreeSum_Sorting(int[] nums, int target) {
    List<Integer[]> result = new ArrayList<>();
    Arrays.sort(nums);
    for (int i = 0; i < nums.length; i++) {
      int left = i + 1;
      int right = nums.length - 1;
      while (left < right) {
        if (nums[i] + nums[left] + nums[right] == target) {
          result.add(new Integer[] { nums[i], nums[left], nums[right] });
          left++;
          right--;
        } else if (nums[i] + nums[left] + nums[right] < target) {
          left++;
        } else {
          right--;
        }
      }
    }
    return result;
  }
}

METHOD 3. Use a Map/Set

Finally, you can also solve the problem using a Map/Set. You just need to iterate through the array, fix the first element, and then try to find the other two elements using the approach similar to the two sum problem.

I’m using a Set in the following solution instead of a Map as used in the two-sum problem because in the two-sum problem, we had to keep track of the index of the elements as well. But In this problem, we just care about the element and not its index.

Time complexity: O(n^2)

import java.util.Set;
import java.util.Scanner;
import java.util.HashSet;

class ThreeSum {

  // Time complexity: O(n^2)
  private static List<Integer[]> findThreeSum(int[] nums, int target) {
    List<Integer[]> result = new ArrayList<>();
    for (int i = 0; i < nums.length; i++) {
      int currentTarget = target - nums[i];
      Set<Integer> existingNums = new HashSet<>();
      for (int j = i + 1; j < nums.length; j++) {
        if (existingNums.contains(currentTarget - nums[j])) {
          result.add(new Integer[] { nums[i], nums[j], currentTarget - nums[j] });
        } else {
          existingNums.add(nums[j]);
        }
      }
    }
    return result;
  }
}

Liked the Article? Share it on Social media!

How to format Date and Time in Java
03
Mar
2021

How to format Date and Time in Java

In this article, you’ll learn how to format Date and Time represented using Date, LocalDate, LocalDateTime, or ZonedDateTime to a readable String in Java.

Format LocalDate using DateTimeFormatter

import java.time.LocalDate;
import java.time.format.DateTimeFormatter;

public class LocalDateFormatExample {
    public static void main(String[] args) {
        DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofPattern("dd/MM/yyyy");

        LocalDate localDate = LocalDate.of(2020, 1, 31);

        System.out.println(localDate.format(dateTimeFormatter));

    }
}
# Output
31/01/2020

Format LocalDateTime using DateTimeFormatter

import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;

public class LocalDateTimeFormatExample {
    public static void main(String[] args) {
        DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofPattern("E, MMM dd yyyy, hh:mm:ss a");

        LocalDateTime localDateTime = LocalDateTime.of(2020, 1, 31, 10, 45, 30);

        System.out.println(localDateTime.format(dateTimeFormatter));

    }
}
# Output
Fri, Jan 31 2020, 10:45:30 AM

Format ZonedDateTime using DateTimeFormatter

import java.time.LocalDateTime;
import java.time.ZoneId;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;

public class ZonedDateTimeFormatExample {
    public static void main(String[] args) {
        DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofPattern("E, MMM dd yyyy, hh:mm:ss a (VV)");

        ZonedDateTime zonedDateTime = ZonedDateTime.of(
                LocalDateTime.of(2020, 1, 31, 10, 30, 45),
                ZoneId.of("America/New_York"));
        
        System.out.println(zonedDateTime.format(dateTimeFormatter));

    }
}
# Output
Fri, Jan 31 2020, 10:30:45 AM (America/New_York)

Format Date and Time using SimpleDateFormat

import java.text.SimpleDateFormat;
import java.util.Date;

public class DateFormatExample {
    public static void main(String[] args) {
        SimpleDateFormat sdf = new SimpleDateFormat("dd/MM/yyyy");

        Date date = new Date();

        System.out.println(sdf.format(date));

    }
}
# Output
24/02/2020

Let’s see another example with a more complex pattern:

import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;

public class DateFormatExample {
    public static void main(String[] args) {
        SimpleDateFormat sdf = new SimpleDateFormat("E, MMM dd yyyy, hh:mm:ss a");

        Calendar calendar = Calendar.getInstance();
        calendar.set(2020, 1, 26, 15, 30, 45);

        Date date = calendar.getTime();

        System.out.println(sdf.format(date));

    }
}
# Output
Wed, Feb 26 2020, 03:30:45 PM
How to delete a directory recursively with all its subdirectories and files in Java
03
Mar
2021

How to delete a directory recursively with all its subdirectories and files in Java

In this short article, you’ll learn how to delete a directory recursively along with all its subdirectories and files.

There are two examples that demonstrate how to achieve this task. The idea behind both of the examples is to traverse the file tree, and delete the files in any directory before deleting the directory itself.

Delete directory recursively – Java 8+

This example makes use of Files.walk(Path) method that returns a Stream<Path> populated with Path objects by walking the file-tree in depth-first order.

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Comparator;

public class DeleteDirectoryRecursively {
    public static void main(String[] args) throws IOException {
        Path dir = Paths.get("java");

        // Traverse the file tree in depth-first fashion and delete each file/directory.
        Files.walk(dir)
                .sorted(Comparator.reverseOrder())
                .forEach(path -> {
                    try {
                        System.out.println("Deleting: " + path);
                        Files.delete(path);
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                });
    }
}

Delete directory recursively – Java 7

The following example uses Files.walkFileTree(Path, FileVisitor) method that traverses a file tree and invokes the supplied FileVisitor for each file.

We use a SimpleFileVisitor to perform the delete operation.

import java.io.IOException;
import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;

public class DeleteDirectoryRecursively1 {
    public static void main(String[] args) throws IOException {
        Path dir = Paths.get("java");

        // Traverse the file tree and delete each file/directory.
        Files.walkFileTree(dir, new SimpleFileVisitor<Path>() {
            @Override
            public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
                System.out.println("Deleting file: " + file);
                Files.delete(file);
                return FileVisitResult.CONTINUE;
            }

            @Override
            public FileVisitResult postVisitDirectory(Path dir, IOException exc) throws IOException {
                System.out.println("Deleting dir: " + dir);
                if (exc == null) {
                    Files.delete(dir);
                    return FileVisitResult.CONTINUE;
                } else {
                    throw exc;
                }
            }
        });
    }
}
How to compare Date and Time in Java
03
Mar
2021

How to compare Date and Time in Java

In this article, you’ll learn several ways of comparing Dates, Times, DateTimes, and DateTimes in different TimeZones in Java.

We use the DateTime API introduced in Java 8 like LocalDate, LocalTime, LocalDateTime, ZonedDateTime, as well as older classes like Date, and Calendar to demonstrate how to compare dates.

Compare dates in Java using LocalDate

import java.time.LocalDate;

public class CompareLocalDateExample {
    public static void main(String[] args) {
        LocalDate date1 = LocalDate.now();
        LocalDate date2 = LocalDate.of(2020, 1, 31);

        // isAfter() method
        if(date1.isAfter(date2)) {
            System.out.println(date1 + " is after " + date2);
        }

        // isBefore() method
        if(date1.isBefore(date2)) {
            System.out.println(date1 + " is before " + date2);
        }

        // isEqual() method
        if(date1.isEqual(date2)) {
            System.out.println(date1 + " is equal to " + date2);
        }

        // compareTo() method
        int diff = date1.compareTo(date2);
        if(diff > 0) {
            System.out.println(date1 + " is greater than " + date2);
        } else if (diff < 0) {
            System.out.println(date1 + " is less than " + date2);
        } else {
            System.out.println(date1 + " is equal to " + date2);
        }
    }
}

Compare date and time in Java using LocalDateTime

import java.time.LocalDateTime;

public class CompareLocalDateTimeExample {
    public static void main(String[] args) {
        LocalDateTime date1 = LocalDateTime.now();
        LocalDateTime date2 = LocalDateTime.of(2020, 1, 31, 10, 15, 45);

        // isAfter() method
        if(date1.isAfter(date2)) {
            System.out.println(date1 + " is after " + date2);
        }

        // isBefore() method
        if(date1.isBefore(date2)) {
            System.out.println(date1 + " is before " + date2);
        }

        // isEqual() method
        if(date1.isEqual(date2)) {
            System.out.println(date1 + " is equal to " + date2);
        }

        // compareTo() method
        int diff = date1.compareTo(date2);
        if(diff > 0) {
            System.out.println(date1 + " is greater than " + date2);
        } else if (diff < 0) {
            System.out.println(date1 + " is less than " + date2);
        } else {
            System.out.println(date1 + " is equal to " + date2);
        }
    }
}

Compare date and time with different TimeZones in Java using ZonedDateTime

import java.time.LocalDateTime;
import java.time.ZoneId;
import java.time.ZonedDateTime;

public class CompareZonedDateTimeExample {
    public static void main(String[] args) {
        ZonedDateTime date1 = ZonedDateTime.of(
                LocalDateTime.of(2020, 1, 31, 10, 30, 45),
                ZoneId.of("America/New_York"));
        ZonedDateTime date2 = ZonedDateTime.of(
                LocalDateTime.of(2020, 1, 31, 10, 30, 45),
                ZoneId.of("Europe/Paris"));

        // isAfter() method
        if(date1.isAfter(date2)) {
            System.out.println(date1 + " is after " + date2);
        }

        // isBefore() method
        if(date1.isBefore(date2)) {
            System.out.println(date1 + " is before " + date2);
        }

        // isEqual() method
        if(date1.isEqual(date2)) {
            System.out.println(date1 + " is equal to " + date2);
        }

        // compareTo() method
        int diff = date1.compareTo(date2);
        if(diff > 0) {
            System.out.println(date1 + " is greater than " + date2);
        } else if (diff < 0) {
            System.out.println(date1 + " is less than " + date2);
        } else {
            System.out.println(date1 + " is equal to " + date2);
        }
    }
}

Compare time in Java using LocalTime

import java.time.LocalTime;

public class CompareLocalTimeExample {
    public static void main(String[] args) {
        LocalTime time1 = LocalTime.of(15, 20, 40);
        LocalTime time2 = LocalTime.of(10, 30, 50);

        // isAfter() method
        if(time1.isAfter(time2)) {
            System.out.println(time1 + " is after " + time2);
        }

        // isBefore() method
        if(time1.isBefore(time2)) {
            System.out.println(time1 + " is before " + time2);
        }

        // compareTo() method
        int diff = time1.compareTo(time2);
        if(diff > 0) {
            System.out.println(time1 + " is greater than " + time2);
        } else if (diff < 0) {
            System.out.println(time1 + " is less than " + time2);
        } else {
            System.out.println(time1 + " is equal to " + time2);
        }
    }
}

Compare date using Date class

import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;

public class CompareDateExample {
    public static void main(String[] args) throws ParseException {

        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");

        Date date1 = sdf.parse("2020-01-31 10:30:45");
        Date date2 = sdf.parse("2020-10-10 14:20:00");

        // after() method
        if(date1.after(date2)) {
            System.out.println(date1 + " is after " + date2);
        }

        // before() method
        if(date1.before(date2)) {
            System.out.println(date1 + " is before " + date2);
        }

        // equals() method
        if(date1.equals(date2)) {
            System.out.println(date1 + " is equal to " + date2);
        }

        // compareTo() method
        int diff = date1.compareTo(date2);
        if(diff > 0) {
            System.out.println(date1 + " is greater than " + date2);
        } else if (diff < 0) {
            System.out.println(date1 + " is less than " + date2);
        } else {
            System.out.println(date1 + " is equal to " + date2);
        }
    }
}

Compare date using Calendar class

import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;

public class CompareCalendarExample {
    public static void main(String[] args) throws ParseException {

        Calendar cal1 = Calendar.getInstance();
        cal1.set(2020, 2, 20, 10, 30, 45);

        Calendar cal2 = Calendar.getInstance();
        cal2.set(2020, 12, 18, 14, 22, 30);


        // after() method
        if(cal1.after(cal2)) {
            System.out.println(cal1 + " is after " + cal2);
        }

        // before() method
        if(cal1.before(cal2)) {
            System.out.println(cal1 + " is before " + cal2);
        }

        // equals() method
        if(cal1.equals(cal2)) {
            System.out.println(cal1 + " is equal to " + cal2);
        }

        // compareTo() method
        int diff = cal1.compareTo(cal2);
        if(diff > 0) {
            System.out.println(cal1 + " is greater than " + cal2);
        } else if (diff < 0) {
            System.out.println(cal1 + " is less than " + cal2);
        } else {
            System.out.println(cal1 + " is equal to " + cal2);
        }
    }
}
Introduction to Data Classes in Kotlin
07
Mar
2021

Introduction to Data Classes in Kotlin

While building any application, we often need to create classes whose primary purpose is to hold data/state. These classes generally contain the same old boilerplate code in the form of getterssettersequals()hashcode() and toString() methods.

Motivation

Consider the following example of a Customer class in Java that just holds data about a Customer and doesn’t have any functionality whatsoever –

public class Customer {
    private String id;
    private String name;

    public Customer(String id, String name) {
        this.id = id;
        this.name = name;
    }

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        Customer customer = (Customer) o;

        if (id != null ? !id.equals(customer.id) : customer.id != null) return false;
        return name != null ? name.equals(customer.name) : customer.name == null;
    }

    @Override
    public int hashCode() {
        int result = id != null ? id.hashCode() : 0;
        result = 31 * result + (name != null ? name.hashCode() : 0);
        return result;
    }
}

You see, for creating a Simple class with only two member fields, we had to write almost 50 lines of code.

Yes, I know that you don’t need to write that code yourself and any good IDE can generate all that boilerplate code for you.

But that code will still be there in your source file and clutter it. Moreover, whenever you add a new member field to the Class, you’ll need to regenerate/modify the constructors, getters/setters and equals()/hashcode() methods.

You can also use a third party library like Project Lombok to generate getters/settersequals()/hashCode()toString() methods and more. But there is no out of the box solution without any library that can help us avoid these boilerplate codes in our application.

Kotlin Data Classes

Kotlin has a better solution for classes that are used to hold data/state. It’s called a Data Class. A Data Class is like a regular class but with some additional functionalities.

With Kotlin’s data classes, you don’t need to write/generate all the lengthy boilerplate code yourself. The compiler automatically generates a default getter and setter for all the mutable properties, and a getter (only) for all the read-only properties of the data class. Moreover, It also derives the implementation of standard methods like equals()hashCode() and toString() from the properties declared in the data class’s primary constructor.

For example, The Customer class that we wrote in the previous section in Java can be written in Kotlin in just one line –

data class Customer(val id: Long, val name: String)

Accessing the properties of the data class

The following example shows how you can access the properties of the data class –

val customer = Customer(1, "Sachin")

// Getting a property
val name = customer.name

Since all the properties of the Customer class are immutable, there is no default setter generated by the compiler. Therefore, If you try to set a property, the compiler will give an error –

// Setting a Property

// You cannot set read-only properties
customer.id = 2	// Error: Val cannot be assigned

Let’s now see how we can use the equals()hashCode(), and toString() methods of the data class-

1. Data class’s equals() method

val customer1 = Customer(1, "John")
val customer2 = Customer(1, "John")

println(customer1.equals(customer2))  // Prints true

You can also use Kotlin’s Structural equality operator == to check for equality. The == operator internally calls the equals() method –

println(customer1 == customer2)  // Prints true

2. Data class’s toString() method

The toString() method converts the object to a String in the form of "ClassName(field1=value1, field2=value)" –

val customer = Customer(2, "Robert")
println("Customer Details : $customer")
# Output
Customer Details : Customer(id=2, name=Robert)

3. Data class’s hashCode() method

val customer = Customer(2, "Robert")
println("Customer HashCode : ${customer.hashCode()}") // Prints -1841845792

Apart from the standard methods like equals()hashCode() and toString(), Kotlin also generates a copy() function and componentN() functions for all the data classes. Let’s understand what these functions do and how to use them –

Data Classes and Immutability: The copy() function

Although the properties of a data class can be mutable (declared using var), It’s strongly recommended to use immutable properties (declared using val) so as to keep the instances of the data class immutable.

Immutable objects are easier to work with and reason about while working with multi-threaded applications. Since they can not be modified after creation, you don’t need to worry about concurrency issues that arise when multiple threads try to modify an object at the same time.

Kotlin makes working with immutable data objects easier by automatically generating a copy() function for all the data classes. You can use the copy() function to copy an existing object into a new object and modify some of the properties while keeping the existing object unchanged.

The following example shows how copy() function can be used –

val customer = Customer(3, "James")

/* 
   Copies the customer object into a separate Object and updates the name. 
   The existing customer object remains unchanged.
*/
val updatedCustomer = customer.copy(name = "James Altucher")
println("Customer : $customer")
println("Updated Customer : $updatedCustomer")
# Output
Customer : Customer(id=3, name=James)
Updated Customer : Customer(id=3, name=James Altucher)

Data Classes and Destructuring Declarations: The componentN() functions

Kotlin also generates componentN() functions corresponding to all the properties declared in the primary constructor of the data class.

For the Customer data class that we defined in the previous section, Kotlin generates two componentN() functions – component1() and component2() corresponding to the id and name properties –

val customer = Customer(4, "Joseph")

println(customer.component1()) // Prints 4
println(customer.component2()) // Prints "Joseph"

The component functions enable us to use the so-called Destructuring Declaration in Kotlin. The Destructuring declaration syntax helps you destructure an object into a number of variables like this –

val customer = Customer(4, "Joseph")

// Destructuring Declaration
val (id, name) = customer
println("id = $id, name = $name") // Prints "id = 4, name = Joseph"

Requirements for Data Classes

Every Data Class in Kotlin needs to fulfill the following requirements –

  • The primary constructor must have at least one parameter
  • All the parameters declared in the primary constructor need to be marked as val or var.
  • Data classes cannot be abstract, open, sealed or inner.

Conclusion

Data classes help us avoid a lot of common boilerplate code and make the classes clean and concise. In this article, you learned how data classes work and how to use them. I hope you understood the all the concepts presented in this article.

Spring Boot Logging Best Practices Guide
05
May
2021

Spring Boot Logging Best Practices Guide

Logging in Spring Boot can be confusing, and the wide range of tools and frameworks make it a challenge to even know where to start. This guide talks through the most common best practices for Spring Boot logging and gives five key suggestions to add to your logging tool kit.

What’s in the Spring Boot Box?

The Spring Boot Starters all depend on spring-boot-starter-logging. This is where the majority of the logging dependencies for your application come from. The dependencies involve a facade (SLF4J) and frameworks (Logback). It’s important to know what these are and how they fit together.

SLF4J is a simple front-facing facade supported by several logging frameworks. It’s main advantage is that you can easily switch from one logging framework to another. In our case, we can easily switch our logging from Logback to Log4j, Log4j2 or JUL.

The dependencies we use will also write logs. For example, Hibernate uses SLF4J, which fits perfectly as we have that available. However, the AWS SDK for Java uses Apache Commons Logging (JCL). Spring-boot-starter-logging includes the necessary bridges to ensure those logs are delegated to our logging framework out of the box.

SLF4J usage:

At a high level, all the application code has to worry about is:

  1. Getting an instance of an SLF4J logger (Regardless of the underlying framework):
    private static final Logger LOG = LoggerFactory.getLogger(MyClass.class);Copy
  2. Writing some logs:
    LOG.info(“My message set at info level”);Copy

Logback or Log4j2?

Spring Boot’s default logging framework is Logback. Your application code should interface only with the SLF4J facade so that it’s easy to switch to an alternative framework if necessary.

Log4j2 is newer and claims to improve on the performance of Logback. Log4j2 also supports a wide range of appenders so it can log to files, HTTP, databases, Cassandra, Kafka, as well as supporting asynchronous loggers. If logging performance is of high importance, switching to log4j2 may improve your metrics. Otherwise, for simplicity, you may want to stick with the default Logback implementation.

This guide will provide configuration examples for both frameworks.

Want to use log4j2? You’ll need to exclude spring-boot-starter-logging and include spring-boot-starter-logging-log4j2.

spring boot logging frameworks

5 Tips for Getting the Most Out of Your Spring Boot Logging

With your initial set up out of the way, here are 5 top tips for spring boot logging.

1. Configuring Your Log Format

Spring Boot Logging provides default configurations for logback and log4j2. These specify the logging level, the appenders (where to log) and the format of the log messages.

For all but a few specific packages, the default log level is set to INFO, and by default, the only appender used is the Console Appender, so logs will be directed only to the console.

The default format for the logs using logback looks like this:

logback default logging format

Let’s take a look at that last line of log, which was a statement created from within a controller with the message “My message set at info level”.

It looks simple, yet the default log pattern for logback seems “off” at first glance. As much as it looks like it could be, it’s not regex, it doesn’t parse email addresses, and actually, when we break it down it’s not so bad.

%clr(%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}){faint}
%clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint}
%clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint}
%m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}Copy

Understanding the Default Logback Pattern

The variables that are available for the log format allow you to create meaningful logs, so let’s look a bit deeper at the ones in the default log pattern example.Show 102550100 entriesSearch:

Pattern PartWhat it Means
%clr%clr specifies a colour. By default, it is based on log levels, e.g, INFO is green. If you want to specify specific colours, you can do that too.

The format is:
%clr(Your message){your colour}

So for example, if we wanted to add “Demo” to the start of every log message, in green, we would write:
%clr(Demo){green}
%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}%d is the current date, and the part in curly braces is the format. ${VARIABLE}:-default is a way of specifying that we should use the $VARIABLE environment variable for the format, if it is available, and if not, fall back to default. This is handy if you want to override these values in your properties files, by providing arguments, or by setting environment variables.

In this example, the default format is yyyy-MM-dd HH:mm:ss.SSS unless we specify a variable named LOG_DATEFORMAT_PATTERN. In the logs above, we can see 2020-10-19 10:09:58.152 matches the default pattern, meaning we did not specify a custom LOG_DATEFORMAT_PATTERN.
${LOG_LEVEL_PATTERN:-%5p}Uses the LOG_LEVEL_PATTERN if it is defined, else will print the log level with right padding up to 5 characters (E.g “INFO” becomes “INFO “ but “TRACE” will not have the trailing space). This keeps the rest of the log aligned as it’ll always be 5 characters.
${PID:- }The environment variable $PID, if it exists. If not, space.
tThe name of the thread triggering the log message.
loggerThe name of the logger (up to 39 characters), in our case this is the class name.
%mThe log message.
%nThe platform-specific line separator.
%wExIf one exists, wEx is the stack trace of any exception, formatted using Spring Boot’s ExtendedWhitespaceThrowableProxyConverter.

Showing 1 to 9 of 9 entriesPreviousNext

Customising the log format

You can customise the ${} variables that are found in the logback-spring.xml by passing in properties or environment variables. For example, you may set logging.pattern.console to override the whole of the console log pattern. 

However, for more control, including adding additional appenders, it is recommended to create your logback-spring.xml and place it inside your resources folder. You can do the same with log4j2 by adding log4j2-spring.xml to your resources folder.

Armed with the ability to customise your logs, you should consider adding:

  • Application name.
  • A request ID.
  • The endpoint being requested (E.g /health).

There are a few items in the default log that I would remove unless you have a specific use case for them:

  • The ‘—’ separator.
  • The thread name.
  • The process ID.

With the ability to customise these through the use of the logback-spring.xml or log4j2-spring.xml, the format of your logs is fully within your control.

2. Configuring the Destination for Your Logs (Appenders and Loggers)

An appender is just a fancy name for the part of the logging framework that sends your logs to a particular target. Both frameworks can output to console, over HTTP, to databases, or over a TCP socket, as well as to many other targets. The way we configure the destination for the logs is by adding, removing and configuring these appenders. 

You have more control over which appenders you use, and the configuration of them, if you create your own custom .xml configuration. However, the default logging configuration does make use of environment properties that allow you to override some parts of it, for example, the date format.

Preset configuration for logging to files are available within Spring Boot Logging. You can use the logback configuration with a file appender or the log4j2 configuration with a file appender if you specify logging.file or logging.path in your application properties.

The official docs for logback appenders and log4j2 appenders detail the parameters required for each of the appenders available, and how to configure them in your XML file. One tip for choosing the destination for your logs is to have a plan for rotating them. Writing logs to a file always feels like a great idea, until the storage used for that file runs out and brings down the whole service. 

Log4j and logback both have a RollingFileAppender which handles rotating these log files based on file size, or time, and it’s exactly that which Spring Boot Logging uses if you set the logging.file property. 

3. Logging as a Cross-Cutting Concern to Keep Your Code Clean (Using Filters and Aspects)

You might want to log every HTTP request your API receives. That’s a fairly normal requirement, but putting a log statement into every controller is unnecessary duplication. It’s easy to forget and make mistakes. A requirement that you want to log every method within your packages that your application calls would be even more cumbersome. 

I’ve seen developers use this style of logging at trace level so that they can turn it on to see exactly what is happening in a production environment. Adding log statements to the start and end of every method is messy, and there is a better way. This is where filters and aspects save the day and avoid the code duplication.

When to Use a Filter Vs When to Use Aspect-Oriented Programming

If you are looking to create log statements related to specific requests, you should opt for using filters, as they are part of the handling chain that your application already goes through for each request. They are easier to write, easier to test and usually more performant than using aspects. If you are considering more cross-cutting concerns, for example, audit logging, or logging every method that causes an exception to be thrown, use AOP. 

Using a Filter to Log Every Request

Filters can be registered with your web container by creating a class implementing javax.servlet.Filter and annotating it with @Component, or adding it as an @Bean in one of your configuration classes. When your spring-boot-starter application starts up, it will create the Filter and register it with the container.

You can choose to create your own Filter, or to use an existing one. To make use of the existing Filter, you need to supply a CommonsRequestLoggingFilter bean and set your logging level to debug. You’ll get something that looks like:

2020-10-27 18:50:50.427 DEBUG 24168 --- [nio-8080-exec-2] o.a.coyote.http11.Http11InputBuffer      : Received [GET /health HTTP/1.1
tracking-header: my-tracking
User-Agent: PostmanRuntime/7.26.5
Accept: */*
Postman-Token: 04a661b7-209c-43c3-83ea-e09466cf3d92
Host: localhost:8080
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
]Copy

If you use the existing one, you have little control over the message that gets logged. 

If you want more control, create your own Filter using this example, and you then have full control over the content of the log message.

Using Aspects for Cross-Cutting Concerns

Aspect-oriented programming enables you to fulfill cross-cutting concerns, like logging for example, in one place. You can do this without your logging code needing to sprawl across every class.

This approach is great for use cases such as:

  • Logging any exceptions thrown from any method within your packages (See @AfterThrowing)
  • Logging performance metrics by timing before/after each method is run (See @Around)
  • Audit logging. You can log calls to methods that have your a custom annotation on, such as adding @Audit. You only need to create a pointcut matching calls to methods with that annotation

Let’s start with a simple example – we want to log the name of every public method that we call within our package, com.example.demo. There are only a few steps to writing an Aspect that will run before every public method in a package that you specify.

  1. Include spring-boot-starter-aop in your pom.xml or build.gradle.
  2. Add @EnableAspectJAutoProxy to one of your configuration classes. This line tells spring-boot that you want to enable AspectJ support.
  3. Add your pointcut, which defines a pattern that is matched against method signatures as they run. You can find more about how to construct your matching pattern in the spring boot documentation for AOP. In our example, we match any method inside the com.example.demo package.
  4. Add your Aspect. This defines when you want to run your code in relation to the pointcut (E.g, before, after or around the methods that it matches). In this example, the @Before annotation causes the method to be executed before any methods that match the pointcut. 

That’s all there is to logging every method call. The logs will appear as:

2020-10-27 19:26:33.269  INFO 2052 --- [nio-8080-exec-2]
com.example.demo.MyAspect                : Called checkHealthCopy

By making changes to your pointcut, you can write logs for every method annotated with a specific annotation. For example, consider what you can do with:

@annotation(com.example.demo.Audit)Copy

4. Applying Context to Your Logs Using MDC

(This would run for every method annotated with a custom annotation, @Audit).

MDC (Mapped Diagnostic Context) is a complex-sounding name for a map of key-value pairs, associated with a single thread. Each thread has its own map. You can add keys/values to the map at runtime, and then reference the keys from that map in your logging pattern. 

The approach comes with a warning that threads may be reused, and so you’ll need to make sure to clear your MDC after each request to avoid your context leaking from one request to the next.

MDC is accessible through SLF4J and supported by both Logback and Log4j2, so we don’t need to worry about the specifics of the underlying implementation. 

The MDC section in the SLF4J documentation gives the simplest examples.

Tracking Requests Through Your Application Using Filters and MDC

Want to be able to group logs for a specific request? The Mapped Diagnostic Context (MDC) will help. 

The steps are:

  1. Add a header to each request going to your API, for example, ‘tracking-id’. You can generate this on the fly (I suggest using a UUID) if your client cannot provide one.
  2. Create a filter that runs once per request and stores that value in the MDC.
  3. Update your logging pattern to reference the key in the MDC to retrieve the value.

Using a Filter, this is how you can read values from the request and set them on the MDC. Make sure to clear up after the request by calling MDC.clear(), preferably in a finally block so that it always runs. 

After setting the value on your MDC, just add %X{tracking}  to your logging pattern (Replacing the word “tracking” with the key you have put in MDC) and your logs will contain the value in every log message for that request. 

If a client reports a problem, as long as you can get a unique tracking-id from your client, then you’ll be able to search your logs and pull up every log statement generated from that specific request.

Other use cases that you may want to put into your MDC and include on every log message include:

  • The application version.
  • Details of the request, for example, the path.
  • Details of the logged-in user, for example, the username.

5. Unit Testing Your Log Statements

Why Test Your Logs?

You can unit test your logging code. Too often this is overlooked because the log statements return void. For example, logger.info(“foo”);  does not return a value that you can assert against. 

It’s easy to make mistakes. Log statements usually involve parameters or formatted strings, and it’s easy to put log statements in the wrong place. Unit testing reassures you that your logs do what you expect and that you’re covered when refactoring to avoid any accidental modifications to your logging behaviour.

The Approach to Testing Your Logs

The Problem

SLF4J’s LoggerFactory.getLogger is static, making it difficult to mock. Searching through any outputted log files in our unit tests is error-prone (E.g we need to consider resetting the log files between each unit test). How do we assert against the logs?

The Solution

The trick is to add your own test appender to the logging framework (e.g Logback or Log4j2) that captures the logs from your application in memory, allowing us to assert against the output later. The steps are:

  1. Before each test case, add an appender to your logger.
  2. Within the test, call your application code that logs some output.
  3. The logger will delegate to your test appender.
  4. Assert that your expected logs have been received by your test appender.

Each logging framework has suitable appenders, but referencing those concrete appenders in our tests means we need to depend on the specific framework rather than SLF4J. That’s not ideal, but the alternatives of searching through logged output in files, or implementing our own SLF4J implementation is overkill, making this the pragmatic choice.

Here are a couple of tricks for unit testing using JUnit 4 rules or JUnit 5 extensions that will keep your test classes clean, and reduce the coupling with the logging framework.

Testing Log Statements Using Junit 5 Extensions in Two Steps

JUnit 5 extensions help to avoid code duplicates between your tests. Here’s how to set up your logging tests in two steps:

Step 1: Create your JUnit extension

Create your extension for Logback

Create your extension for Log4j2

Step 2: Use that rule to assert against your log statement with logback or log4j2

Testing Log Statements Using Junit 4 Rules in Two Steps

JUnit 4 rules help to avoid code duplication by extracting the common test code away from the test classes. In our example, we don’t want to duplicate the code for adding a test appender to our logger in every test class.

Step 1: Create your JUnit rule. 

Create your rule for Logback

Create your rule for Log4j2

Step 2: Use that rule to assert against your log statements using logback or log4j2.

With these approaches, you can assert that your log statements have been called with a message and level that you expect. 

Conclusion

The Spring Boot Logging Starter provides everything you need to quickly get started, whilst allowing full control when you need it. We’ve looked at how most logging concerns (formatting, destinations, cross-cutting logging, context and unit tests) can be abstracted away from your core application code.

Any global changes to your logging can be done in one place, and the classes for the rest of your application don’t need to change. At the same time, unit tests for your log statements provide you with reassurance that your log statements are being fired after making any alterations to your business logic.

These are my top 5 tips for configuring Spring Boot Logging. However, when your logging configuration is set up, remember that your logs are only ever as good as the content you put in them. Be mindful of the content you are logging, and make sure you are using the right logging levels.