Monday, November 07, 2016

Java threads Tutorial or Multithreading Tutorial in Java

Java threads Tutorial or Java 8 concurrency Tutorial or Java 8 Threads or Multithreading tutorial in Java


In this article we will discuss about java threads and also the new Java Concurrency features in Java 8..
Some of the topics covered here include are listed below. I have detailed examples for each case 
and also you can download the samples  for all this examples here.

Please click here to Download all the Examples  and import them as Java Projects.
Thread2DemoJava8Features.zip
and
Thread1DemoJava7Features.zip

You need Jdk 8.

Whats is Java thread ?

A thread is different path of execution within a process. A thread is light weight compared to Process.
A process has its own memory space (or to be precise own address space) and within a process you can have many threads and these threads 
all can access the memory of the this process. A process cannot access memory (I mean address space) of other space unless using something like IPC (inter process communication...) Threads share the process's resources, including memory and open files. This makes for efficient, but potentially problematic, communication

Java concurrency Topics or Key Concurrency Topics we will discuss: 

Here is the high level overview of the topics we will cover in this post. Its going to be long article, trust me once you read the entire article you appreciate all the features and how each can be used in different context to solve a particular problem. I will just give a brief overview then we will discuss each topic in depth with a working example.
  • Creating a thread using Thread class and Runnable Interface.
  • Creating a thread using Thread class and Runnable Interface using Java 8 Lambda Syntax.
  • Running multiple threads using "Executor service" and kick start them using
    eg: ExecutorService executor = Executors.newFixedThreadPool(2); 
  • ExecutorService.submit();
  • Callable and Future interfaces: Want your threads run method to return results ?  In case you want to return results from run method use Callable interface and  put your code in call() method (similar to run() but can return a result). Similar to Runnable interface's run method, the Callable interface has has call() which returns result wrapped in Future<?> interface
  • Using ExecutorService.invokeall() to submit multiple Threads that implement Callable<...> interface
  • Using "Executors.newWorkStealingPool()" - where in the number of threads may grow or Shrink.A work-stealing pool makes no guarantees about the order in which submitted tasks are executed and using "executor.invokeAny(callableTasks);" - Instead of returning future objects this method blocks until the first callable terminates and returns the result of that callable.
  • Scheduling when to run a thread: Want to run a callable at a specified time  here are some options.
    • ScheduledFuture<Employee> schedFuture0 = executor.schedule(task0, 3, TimeUnit.SECONDS); //Execute after 3 seconds
    • ScheduledFuture<Employee> schedFuture1 = executor.schedule(task1, 10, TimeUnit.SECONDS); //Execute after 10 seconds
    • ScheduledFuture<Employee> schedFuture2 = executor.schedule(task2, 3, TimeUnit.SECONDS); //Execute after 3 seconds.
  • Want to run a thread with fixed delay ?
    • executor.scheduleWithFixedDelay(task, initialDelay, period, TimeUnit.SECONDS); 
    • executor.scheduleAtFixedRate(task, initialDelay, period, TimeUnit.SECONDS);
  • Synchronized block vs Synchronized method.
  • Calling of wait(), notify(), notifyAll() in a synchronized context
    • wait() method tells the current thread (thread which is executing code inside a synchronized method or lock) to give up monitor and go to waiting state.
    • notify() method Wakes up a single thread that is waiting on this object's monitor.
    • notifyAll() method wakes up all the threads that called wait( ) on the same object.
  • Volatile why a variable should be marked volatile (To tell thread not to cache and instead re-read the value each time.)
  • CountDownLatch
    CountDownLatch is used to start a series of threads and then wait until all of them are complete (or until they call countDown() a given number of times CountDownLatch cannot be reused after meeting the final count.
    (Just opposite of Semaphore discussed further. In case you want to reuse countdownlatch try "CyclicBarrier")
  • CyclicBarrier:
    A synchronization aid that allows a set of threads to all wait for each other to reach a common barrier point.
    CyclicBarriers are useful in programs involving a fixed sized party of threads that must occasionally wait for each other.
    The barrier is called cyclic because it can be re-used after the waiting threads are released.
    ( Cyclicbarrier is similar to countdownlatch except CyclicBarrier can be re used.)
  • Count down Latch VS CyclicBarriers:
    • CountDownLatch can not be reused after meeting the final count.
    • CountDownLatch can not be used to wait for Parallel Threads to finish.
    • CyclicBarrier can be reset thus reused.
    • CyclicBarrier can be used to wait for Parallel Threads to finish.
  • RentrantLock: 
    A re-entrant mutual exclusion lock with the same basic behavior and semantics as the implicit monitor lock accessed using
    a synchronized methods or statements, but with extended capabilities.
    Locks support various methods for finer grained lock control thus are more expressive than implicit monitors.
  • RentrantLock with condition:
    The Condition interface factors out the java.lang.Object monitor methods (wait(), notify(), and notifyAll()) into distinct objects to give the effect of having multiple wait-sets per object,by combining them with the use of arbitrary Lock implementations.Where Lock replaces synchronized methods and statements, Condition replaces Object monitor methods.
    {eg:
          private Lock lock = new ReentrantLock();
          private Condition condition = lock.newCondition();
          condition.await() //Wait for the signal and release the lock kind like wait()
          condition.signal() //Send signal to for threads that are waiting for this condition kind like notify().
    }
  • ReadWriteLock: Many threads can have read lock at same time but only on thread will have write lock.
  • StampedLock: Java 8 ships with a new kind of lock called StampedLock which also support read and write locks just like ReadWrietLocak. In contrast to ReadWriteLock the locking methods of a StampedLock return a stamp represented by a long value.
  • SemaphoresSemaphore is used to control the number of concurrent threads that are using a resource.The Re-entrant locks usually grant exclusive access to variables or resources, a semaphore is capable of maintaining whole sets of permits.Its more like having tokens and once the token is used we need wait for the token to be released so that someone else can use it.
  • Dead LockWhat is a dead lock ? When two threads are waiting on each other for a resource to be available, while each one as acquired lock on other resource.
    e.g: Say we have two lists (list1 and list2).
    In one thread get lock on list1 and list2 in the order I specified. In second thread get lock on list2 and list1 .
    Both threads will never get locks on both objects due to different orders
  • Preventing dead lock in the above case by using "ReentrantLock" by declaring lock per resource and getting all locks at once.
  • Producer/Consumer example using synchronized keyword, wait() and notify() features.
  • Producer/Consumer example using ArrayBlockingQueue: Thread safe and can be accessed by multiple threads. ArrayBlockingQueue infact uses RentrantLock inside it if you see the implementation." If the Queue is full any write calls to Queue will be a blocking call.
    If queue is empty any reads will be blocking call.
  • Interrupting a thread: We can interrupt a thread by calling Thread.currentThread().interrupt().
    • If the current thread is in sleep an interruptedExcpetion will be thrown
    • If current thread is running a flag will be set we can check that using "Thread.currentThread().isInteruppted()"
    • If the thread is blocked on IO then the thread's interrupt status will be set, and the thread will receive a "java.nio.channels.ClosedByInterruptException"
  • AtomicIntegerThread safe. Internally, the atomic classes make heavy use of compare-and-swap (CAS), an atomic instruction directly supported by most modern CPU's. Those instructions usually are much faster than synchronizing via locks.So my advice is to prefer atomic classes over locks in case you just have to change a single mutable variable concurrently.
  • ConcurrentHashMap:Similar to HashTable its Synchronized. ConcurrentHashMap uses multiple buckets to store data. This avoids read locks and greatly improves performance over a HashTable. Both are thread safe, but there are obvious performance wins with ConcurrentHashMap.
    However when you read from a ConcurrentHashMap using get(), there are no locks,contrary to the HashTable for which all operations are simply synchronized. HashTable was released in old versions of Java whereas ConcurrentHashMap is a java 5+ thing.
Examples for each of the scenarios we described above are as follows.

Creating a thread using Thread class and Runnable Interface: 


Tuesday, September 27, 2016

Lambda Expressions Java 8 or Java Lambda Expressions


Lambda Expressions Java 8 ( or Closures)

There are several ways lambda expressions can be written.I went through several blogs and also some Oracle docs.The outcome of that are these samples which shows how to use Lambda expressions.
Java 8 supports functional programming

Lambda expression or Closures allows functional programming, and simplifies the development a lot. A lambda expression is characterized by the following syntax - (parameter) -> expression body.
Lambdas work only on Single Method Abstraction (SAM).
i.e. interfaces which have just one method. Anonymous class can implement any.

Here is an example of anonymous class (java 7) and
same thing which can be written even more easily with Java 8 using lambda.

/* 
 You can read more for difference between Annonymos class and Lambda.
 http://stackoverflow.com/questions/22637900/java8-lambdas-vs-anonymous-classes
 
*/
public class AnonymousClass2 {

 public static void main(String[] args) {
  Thread t = new Thread(runnable1);
  t.start();
  t = new Thread(runnable2);
  t.start();
 }
 
 /*
  * Old way without Lambda: write entire stuff here
  */
 static Runnable runnable1 = new Runnable() {
  @Override
  public void run() {
   System.out.println("Running without Lambda");
  }
 };
 
 
 /* With Lambda one line. Lambdas work only on Single Method Abstraction (SAM). 
  *i.e. interfaces which have just one method. Anonymous class can implement any. */
 
 static Runnable runnable2 = () -> { System.out.println("Running from Lambda"); };
}


Using Lambda Expression for Sorting (Without Predicates) using Collections.sort(.....).
For each call to Collections sort() method we will pass an expression.
There are several ways we can express this.
For this we will use Employee  java object as described here .

package com.rama.jdk8.closures;
import java.util.Date;

public class Employee {
 
 private String fName;
 private String lName;
 private Date date;
 
 public Employee(String fName,String lName,Date date){
  this.fName = fName;
  this.lName = lName;
  this.date = date;
 }
 //....
 // Generate Getters and Setters. Left it intentionally 
 
 @Override
 public String toString() {
  StringBuilder sbud = new StringBuilder();
  sbud.append(" fName="+ fName);
  sbud.append(" lName="+ lName);
  sbud.append(" date="+ date +"\n");
  return sbud.toString();
 }
 
}

Here is the class that shows  the following flavors of sorting.
i.e Sorting using

  • Comparator.comparing(Employee::getfName)
  • By Passing lambda expression (e1,e2)-> e1.getfName().compareTo(e2.getfName())
  • Reversing the order after Sorting using "comparator.reversed()"
  • Nesting of Comparators like Comparator.comparing(Employee::getfName).thenComparing(Employee::getlName);
  • Using Stream and difference stream() and ParealleStream() on a collection.
All the above mentioned are covered in this class. Try experimenting more by commenting and uncommenting the code.

package com.rama.jdk8.closures;

import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.Date;
import java.util.List;
import java.util.function.Consumer;
import java.util.stream.Collectors;

/**
 * @author TWreddy
 *  This is very basic example of how to use Lambdas for sortings. 
 *  Later I will show how to use Predicates to do the sorting
 *  For all this examples we will use Employee.java which is POJO with few setters and getters 
 */
public class EmployeeSortLambda {
 
 public static void main(String[] args) {
 /* Comment or Uncomment the method you want to run  */
  //sortByFirstName();
  //sortByLastName();
  //sortByDob();
  //getByLastName("Reddy");
  //sortEmployeesByFname();
  //sortEmployeesByFnameinRevereseOrder();
  //sortEmployeesByFnameinRevereseOrderAndThenByLastName();
  //parallelSorting();
  printAll();
 }
 
 public static void  sortByFirstName(){
   List<Employee> employeeList = getEmployees();
   //Collections.sort(employeeList,(e1,e2)-> e1.getfName().compareTo(e2.getfName()));
   //Easier way 
   Collections.sort(employeeList,Comparator.comparing(Employee::getfName));
   System.out.println(employeeList);
  }
  
  public static void  sortByLastName(){
   List<Employee> employeeList = getEmployees();
   Collections.sort(employeeList, (e1, e2) -> { return e1.getlName().compareTo(e2.getlName());});
   System.out.println("Sorted by Last Name:\n"+ employeeList);
   }
  
  
  public static void  sortByDob(){
   List<Employee> employeeList = getEmployees();
   
   Collections.sort(employeeList, (e1, e2) -> { 
    return e1.getDate().compareTo(e2.getDate());
   });
   
   System.out.println("Sorted by Date:\n"+ employeeList);
   }
  
  public static void  getByLastName(String lastName){
   List<Employee> employeeList = getEmployees();
   
   List<Employee> employeeByLastName = new ArrayList<Employee>();
   employeeList.forEach((emp) -> {
    if(emp.getlName().equalsIgnoreCase(lastName)){
     employeeByLastName.add(emp);
    }
   });
   
   System.out.println(" Employee by Last name=\n"+ employeeByLastName);
   }
  
  public static void sortEmployeesByFname(){
   List<Employee> employees  = getEmployees();
  //Sort all employees by first name
  employees.sort(Comparator.comparing(e -> e.getfName()));
  //OR you can use below
  employees.sort(Comparator.comparing(Employee::getfName));
  
  System.out.println("Sorty by Fname:\n"+ employees);
      
  }
  
  /**
  *  Reversiong the SOrt order 
  */
 public static void sortEmployeesByFnameinRevereseOrder(){
   List<Employee> employees  = getEmployees();
  //Sort all employees by first name
   Comparator<Employee> comparator = Comparator.comparing(e -> e.getfName());
   employees.sort(comparator.reversed());
   System.out.println("Sort by Fname but in reverse Order:\n"+ employees);
      
  }
  
  /**
  *  Nesting of Comparators to sort using Lambda
  */
 public static void sortEmployeesByFnameinRevereseOrderAndThenByLastName(){
   List<Employee> employees  = getEmployees();
  //Sort all employees by first name
  Comparator<Employee> groupByComparator = Comparator.comparing(Employee::getfName).thenComparing(Employee::getlName);
  employees.sort(groupByComparator);
  System.out.println("Sort by Fname and then by Lname:\n"+ employees);
      
  }
 
  /**
   * Use only if the collections is large else go with normal way
   * Sorting in Parallel.
   **/
  public static void parallelSorting(){
   
   List<Employee> employees  = getEmployees();
   //Sort all employees by first name
   Comparator<Employee> groupByComparator =       Comparator.comparing(Employee::getfName).thenComparing(Employee::getlName);
   //employees.stream().parallel().sorted(groupByComparator).collect(Collectors.<Employee>toList());
   employees.parallelStream().sorted(groupByComparator).collect(Collectors.<Employee>toList());
   System.out.println("Parallel sorting using "+ employees);
  }
  
  /**
  *  Lets say you want to print every Object in the List 
  */
 public static void printAll(){
   List<Employee> employees  = getEmployees();
   //employees.stream().forEach(System.out::println);
   //or
   Consumer<Employee> printEmp= (emp)-> System.out.println(emp);
   employees.stream().forEach(printEmp);
  }
 
  
  public static List<Employee> getEmployees(){
   List<Employee> employeeList = new ArrayList<Employee>();
   
   Employee e1 = new Employee("RamaChandra","Reddy",new Date("11/23/1990"));
   Employee e2 = new Employee("John","Smith",new Date("06/06/1975"));
   Employee e3 = new Employee("Douglas","Whitman",new Date("11/22/1974"));
   Employee e4 = new Employee("Albert","Einstien",new Date("01/01/1950"));
   Employee e5 = new Employee("Rama","Krishnan",new Date("06/14/1975"));
   Employee e6 = new Employee("Sanjeev","Reddy",new Date("01/01/1983"));
   
  
   employeeList.add(e1);
   employeeList.add(e2);
   employeeList.add(e3);
   employeeList.add(e4);
   employeeList.add(e5);
   employeeList.add(e6);
   
   
   return employeeList;
  }
}

Passing Function as Parameter using Lambda Expression

Ever wondered how to pass a function as parameter. We can do that with Java Lambda expressions
Here is an example.

/**
 * 
 * @author twreddy
 * How to use  Function as prameter in Lambda
 **/
public class LambdaFunctionArgument {

 interface Circle {
  double get(double radius); /// Can be area or circumeference...etc.
 }

 public double circleOperation(double radius, Circle c) {
  return c.get(radius);
 }
 
 public static void main(String args[]){
  LambdaFunctionArgument reference = new LambdaFunctionArgument();
  //Circle.get() implementation to find area....
  Circle circleArea =  (double r) -> {  return (Math.PI * r *r);}; 
  //Circle.get() implementation to calculate circumference... 
  Circle circleCircumference =  (double r)->{return (2 * Math.PI * r);}; 
  double area = reference.circleOperation(10, circleArea);
  double circumference = reference.circleOperation(10, circleCircumference);
 
  System.out.println("Area: "+area+" . Circumference: "+circumference);
 }
}
Using Java 8 Predefined functional interfaces (see java.util.function.* package ) provided as part of  Java 8. All interfaces in this package are annotated with @FunctionalInterface  and have Single Abstract Method (SAM) .
Here is one example you can try others.

import java.util.function.DoubleBinaryOperator;

public class MathOperationLambda {
 
 public static void main(String[] args) {
  
  MathOperation addOperation = (a,b)->{ return a+b;};
  MathOperation subtractOperation = (a,b)->{ return a-b;};
  MathOperation divideOperation = (a,b)->{ return a/b;};
  MathOperation multiplyOperation = (a,b)->{ return a*b;};
  
  float a =10, b=5;
  System.out.println(addOperation.operation(a,b));
  System.out.println(subtractOperation.operation(a,b));
  System.out.println(divideOperation.operation(a,b));
  System.out.println(multiplyOperation.operation(a,b));
  
  
  //Or you can use predefined functional interface 
  DoubleBinaryOperator addOp = (x,y)->{ return x+y;};
  System.out.println(addOp.applyAsDouble(a,b));
  
 }
 
}

interface MathOperation{
 public float operation(float a, float b);
}



Java 8 Streams:

Supports functional-style operations on streams of elements, such as map-reduce transformations n collections. Use stream operations to express sophisticated data processing queries.
Typical processing patterns on collections are similar to SQL-like operations such as “finding” (a highest value or lowest value)  or “grouping”  or doing count...etc (More like SQL operations on Collection and more..)

Here is one example of using stream()  vs  parallelStream()

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.TimeUnit;

/**
 * @author twreddy
 *   Good example showing usage of stream() and parallelStream() and the difference 
 *   And also some new methods thats added to Map class.
 */
public class SequentialParallelSort {
 
 public static void main(String[] args) {
  List<String> values = getData();
  
  parallelSort(values);
  sequentialSort(values);
 }
 
 public static void parallelSort(List<String> values){
  long t0 = System.nanoTime();
  long count = values.parallelStream().sorted().count();
  System.out.println(count);
  long t1 = System.nanoTime();
  long millis = TimeUnit.NANOSECONDS.toMillis(t1 - t0);
  System.out.println(String.format("parallel sort took: %d ms", millis));
 }
 
 public static void sequentialSort(List<String> values){
  long t0 = System.nanoTime();
  long count = values.stream().sorted().count();
  System.out.println(count);
  long t1 = System.nanoTime();
  long millis = TimeUnit.NANOSECONDS.toMillis(t1 - t0);
  System.out.println(String.format("sequential sort took: %d ms", millis));
 }
 
 public static List<String> getData(){
  int max = 1000000;
  List<String> values = new ArrayList<>(max);
  for (int i = 0; i < max; i++) {
      UUID uuid = UUID.randomUUID();
      values.add(uuid.toString());
  }
  return values;
 }
 
}


JDK 8 Optional:  How to handle/avoid  Nullpointer Exceptions ?
=======================================================
JDK 8 has java.util.Optional which can be used (It can avoid Null pointer Exception in most cases, yes not all cases)

Here is one simple example.

Another great example at 
http://www.oracle.com/technetwork/articles/java/java8-optional-2175753.html


import java.util.Optional;
public class OptionalTest {

 public static void main(String[] args) {
  
  Optional<Integer> canBeEmpty1 = Optional.of(5);
  System.out.println(canBeEmpty1.isPresent());                    // returns true
  System.out.println(canBeEmpty1.get());                          // returns 5

  Optional<Integer> canBeEmpty2 = Optional.empty();
  System.out.println(canBeEmpty2.isPresent());                    // returns false
  
  Optional<String> optional = Optional.of("bam");
  System.out.println(optional.isPresent());           // true
  System.out.println(optional.get());                 // "bam"
  System.out.println(optional.orElse("fallback"));    // "bam"
  optional.ifPresent((s) -> System.out.println(s.charAt(0)));     
 }

}

Java 8 Predicates: 

This is a functional interface and can therefore be used as the assignment target for a lambda
expression or method reference.You can use them anywhere where you need to evaluate a condition
on group/collection of similar objects such that evaluation can result either in true or false

@FunctionalInterface
public interface Predicate<T>{
....
}


Lets use our previous Employee object 
public class Employee {
    
    private Integer id;
    private Integer age;
    private String gender;
    private String firstName;
    private String lastName;
    
    public Employee(Integer id, Integer age, String gender, String fName, String lName){
        this.id = id;
        this.age = age;
        this.gender = gender;
        this.firstName = fName;
        this.lastName = lName;
    }
      
          //...
  //... Generate Getters and Setters.....Please.
 
  @Override
     public String toString() {
         return this.id.toString()+" - "+this.age.toString() +" Gender:"+ this.getGender() +"\n"; 
     }
}
//Here are the Predicates we will define to SORT.

Also you will see that we have used a method called flatMap() and there is another method map() which is not mentioned in this example.
Both map and flatMap can be applied to a Stream<T> and they both return a Stream<R>. The difference is that the map operation produces one output value for each input value, whereas the flatMap operation produces an arbitrary number (zero or more) values for each input value.


import java.util.List;
import java.util.function.Predicate;
import java.util.stream.Collectors;

/** 
 * Lets define the Predicates which we will use for Sorting the Employees collection.
 */
public class EmployeePredicates {
 
 public static Predicate<Employee> isAdultMale() {
  return emp -> emp.getAge() > 21 && emp.getGender().equalsIgnoreCase("M");
  
 }
 
  public static Predicate<Employee> isAdultFemale(){
   return emp -> emp.getAge() > 21 && emp.getGender().equalsIgnoreCase("F");
  }
  
  public static Predicate<Employee> isAgeMoreThan(Integer age){
   return emp-> emp.getAge() > age;
  }
  
  public static List<Employee> filterEmployees(List<Employee>  employees,Predicate<Employee> predicate){
   return employees.stream().filter(predicate).collect(Collectors.<Employee>toList());
  }
}
Now lets put the Predicates we defined to test. (Also compare against the sorting technique we used earlier using Lambda expression on employee collection without Predicates)
You clearly see this more cleaners.

mport static com.rama.jdk8.predicate.EmployeePredicates.*;
public class EmployeePredicateTest {
 
 
 public static void main(String[] args) {
  
         Employee e1 = new Employee(1,23,"M","Rick","Beethovan");
         Employee e2 = new Employee(2,13,"F","Martina","Hengis");
         Employee e3 = new Employee(3,43,"M","Ricky","Martin");
         Employee e4 = new Employee(4,26,"M","Jon","Lowman");
         Employee e5 = new Employee(5,50,"F","Cristine","Maria");
         Employee e6 = new Employee(6,15,"M","David","Feezor");
         Employee e7 = new Employee(7,68,"F","Melissa","Roy");
         Employee e8 = new Employee(8,79,"M","Alex","Gussin");
         Employee e9 = new Employee(9,15,"F","Neetu","Singh");
         Employee e10 = new Employee(10,45,"M","Naveen","Jain");
         
         List<Employee> employeeList = new ArrayList<Employee>();
         employeeList.addAll(Arrays.asList(e1,e2,e3,e4,e5,e6,e7,e8,e9,e10));
         System.out.println(employeeList);
         
          
         System.out.println("Male Adults="+ filterEmployees(employeeList,isAdultMale()));
System.out.println("Not Female Adults="+ filterEmployees(employeeList,isAdultFemale().negate()));
         System.out.println("Female Adult="+ filterEmployees(employeeList,isAdultFemale()));
         System.out.println("Age > 40 ="+ filterEmployees(employeeList,isAgeMoreThan(40)));
         System.out.println("Adult male and age > 40="+ filterEmployees(employeeList,isAdultMale().and(isAgeMoreThan(40))));
         
 }
}

Further Examples of Lambda :

Here is order of operations you can apply on collections  once you obtain data as Stream 


List<Integer> empIds = 
    employeeList.parallelStream()
                .filter(isAdultMale())
                .sorted(comparing(Employee::getId).reversed())
                .map(Employee::getId)
                .collect(toList());
Using Lambda expression you can simulate functionality like 
- "ORACLE LIKE function." - We can use  filter(....)
- "GROUP BY like in oracle." - We can use Collectors.groupingBy(...)
          
More examples..

For this lets use a DTO called Article.java as below.

 public class Article {
 private  String title;
 private  String author;
 private  List<String> tags = new ArrayList<String>();
 
 public Article(){
  
 }
 
 public Article(String title,String author, List<String> tags){
  this.title = title;
  this.author = author;
  this.tags.addAll(tags);
 }
 
 //.....
 //... Please generate getters and Setters. 
 @Override
 public String toString() {
  StringBuilder sbud = new StringBuilder();
  sbud.append("title="+title);
  sbud.append(" author="+author);
  sbud.append(" tags="+ tags.toString());
  return sbud.toString();
 }
  
}

And lets use the above DTO and Lambda expressions now.

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;

/**
 * @author twreddy
 *  Streams : Lambda expressions enable you to do this,
 *  to treat functionality as method argument, or code as data.
 *  
 */
public class ArticleTest {

 private List<Article> articles = new ArrayList<Article>();
 
  public static void main(String[] args) {
  ArticleTest at = new ArticleTest();
   at.loadData();
  System.out.println("getAllJavaArticle = "+at.getAllJavaArticle());
  System.out.println("getFirstJavaArticle="+at.getFirstJavaArticle());
  System.out.println("groupByAuthor="+ at.groupByAuthor());
  System.out.println("getDistinctTags="+ at.getDistinctTags());
 }
 
 
 /**
  * LIKE ORACLE LIKE function.
  * @return
  */
 public Optional<Article> getFirstJavaArticle() {  
     return articles.stream()
         .filter(article -> article.getTags().contains("Java"))
         .findFirst();
   }

 
 public List<Article> getAllJavaArticle() {  
      return articles.stream()
         .filter(article -> article.getTags().contains("Java")).collect(Collectors.toList());
   }

 /* OLD way  JDK 7 */
 public Map<String, List<Article>> groupByAuthorJava7() {

     Map<String, List<Article>> result = new HashMap<>();

     for (Article article : articles) {
         if (result.containsKey(article.getAuthor())) {
             result.get(article.getAuthor()).add(article);
         } else {
             ArrayList<Article> articles = new ArrayList<>();
             articles.add(article);
             result.put(article.getAuthor(), articles);
         }
     }

     return result;
 }
 
 /*
  * GROUP BY like in oracle.
  */
 public Map<String, List<Article>> groupByAuthor() {  
     return articles.stream()
         .collect(Collectors.groupingBy(Article::getAuthor));
 }    
 
 /*
  * Like to Extract one property so use flatmap.
  */
 public Set<String> getDistinctTags() {  
     return articles.stream()
         .flatMap(article -> article.getTags().stream())
         .collect(Collectors.toSet());
 }
 
 
 public List<Article> loadData(){
  List<String> l1 = new ArrayList<String>();
  l1.add("Java");
  Article a1 = new Article("Complete reference","Herbert Schildt",l1);
  
  List<String> l2 = new ArrayList<String>();
  l2.add("Algorithms");
  Article a2 = new Article("Datastructures and Algorithms","Padma Reddy",l2);
  
  List<String> l3 = new ArrayList<String>();
  l3.add("Algorithms");
  Article a3 = new Article("Finite automation","Padma Reddy",l3);
  articles.add(a1);
  articles.add(a2);
  articles.add(a3);
  return articles;
  
 }
 
}

More usage of collections.stream()
=============================

import java.util.Arrays;
import java.util.List;
import java.util.function.Predicate;
import java.util.stream.Collectors;

import com.rama.jdk8.predicate.Employee;

public class BaseStreamTest {
 
 public static void main(String[] args) {
  
  
  List<String> list = Arrays.asList("abc", "", "bc", "efg", "abcd","", "jkl",null);
  
  //As good as writing a predicate which checks that Stirng is not null.
  List<String> filtered = list.stream().filter(string -> (string !=null) && !string.isEmpty()).collect(Collectors.toList());
  System.out.println("filtered with out predicate using plain lambda expression ="+ filtered);
  
  List<String> filteredWithPredicate = list.stream().filter(isNotNull()).collect(Collectors.toList());
  System.out.println("filteredWithPredicate="+ filteredWithPredicate);
  
  List<String> filteredWithPredicateNulls = list.stream().filter(isNull()).collect(Collectors.toList());
  System.out.println("filteredWithPredicateNulls="+ filteredWithPredicateNulls);
  
 }
 
  public static Predicate<String> isNotNull() {
   return str -> (str != null) && !str.isEmpty();
  }
  
  public static Predicate<String> isNull(){
   return str -> ((str == null)||(str.isEmpty()));
  }

}

At lastly there some new methods on Map class in JDK 8

//Map Class in JdK has some new methods,.  
Map<Integer, String> map = new HashMap<>();
map.putIfAbsent(1, "Rama");
map.putIfAbsent(1, "Chandra");
map.forEach((id, val) -> System.out.println(val));
Code samples can be downloaded at
Lamda Examples Code Download

How to connect LDAP server using Spring framework ? or How to perform search operation in ldap using spring ?



In the following post we will look in to

  • How to connect LDAP server using Spring framework 
  • Simplify directory access with Spring LDAP
  • Using Spring LdapTemplate.
  • How to perform search operation in ldap using spring ?
  • Introduction to Spring LDAP ?
  • How to use Spring with LDAP ?


First I will show a simple example of how to connect to LDAP using plain java that way you will appreciate Spring LDAP template even better.
There are several threads online on this subject. I adding this article so that i can provide more details on how to use 
spring provide LDAP filters. 

Ok lets start with simple Java approach first.
A simple interface that returns users we are looking for as list of Strings or as list of java objects.
Here is the code to do that lets call this class TraditionalLDAPUserRepoImpl.java and the interface as LDAPUserRepo .java

Interface LDAPUserRepo.java
package com.rama.util.activedirectory;
import java.util.List;

public interface  LDAPUserRepo {
 
 /*  Just return usernames that matches your LDAP query criteria. Just as list of Strings*/
 public List<String> getAllPersonNames();
 
 /* This method will return list of java objects  with user name, email etc */
 public List<LDAPUser> getAllPersonNameObjs();
}
Code that show implementation of TraditionalLDAPUserRepoImpl.java
Needless to say you need to change you ldap address and credentials.
// A simple lookup using plain Java 
package com.rama.util.activedirectory;

import java.util.Hashtable;
import java.util.LinkedList;
import java.util.List;

import javax.naming.Context;
import javax.naming.NameNotFoundException;
import javax.naming.NamingEnumeration;
import javax.naming.NamingException;
import javax.naming.directory.Attribute;
import javax.naming.directory.Attributes;
import javax.naming.directory.DirContext;
import javax.naming.directory.InitialDirContext;
import javax.naming.directory.SearchControls;
import javax.naming.directory.SearchResult;

public class TraditionalLDAPUserRepoImpl implements LDAPUserRepo {

 public List<String> getAllPersonNames() {
  Hashtable<String, String> authEnv = new Hashtable<String, String>();
  authEnv.put(Context.INITIAL_CONTEXT_FACTORY,"com.sun.jndi.ldap.LdapCtxFactory");
  //https://docs.oracle.com/javase/8/docs/technotes/guides/jndi/jndi-ldap-gl.html
  authEnv.put(Context.PROVIDER_URL, "ldap://dev.rama.int:389/DC=DevS1,DC=int");
     authEnv.put(Context.SECURITY_AUTHENTICATION, "simple");
  //Your  security principla could be different
     authEnv.put(Context.SECURITY_PRINCIPAL,"CN=rama,OU=Service_Account,DC=DevS1,DC=int");
     authEnv.put(Context.SECURITY_CREDENTIALS,"test123");
     
     //https://docs.oracle.com/javase/8/docs/technotes/guides/jndi/jndi-ldap-gl.html
     //authEnv.put("java.naming.ldap.referral.limit","10");
     //https://docs.oracle.com/javase/jndi/tutorial/ldap/referral/jndi.html
     //authEnv.put(Context.REFERRAL,"follow");
     //For example, the following code specifies that the provider should block until 24 entries have been read from the server or until the enumeration terminates, whichever produces the fewer number of results:
     //authEnv.put(Context.BATCHSIZE,"400");
      
  DirContext ctx;
  try {
   ctx = new InitialDirContext(authEnv);
  } catch (NamingException e) {
   throw new RuntimeException(e);
  }

  List<String> list = new LinkedList<String>();
  NamingEnumeration results = null;
  try {
   SearchControls controls = new SearchControls();
   controls.setSearchScope(SearchControls.SUBTREE_SCOPE);
   
   /*
    If you comment out the below line all properties will  be pulled.
    Here just to show I am explicitly mentioning the properties we want.
    sn- Simple name
    CN- complete name
    OU - organizational unit.
    
    Think of this as a oracle select statement with the columns you wanted listed in select clause 
   */
   controls.setReturningAttributes(new String[] {"sAMAccountName","givenName", "sn","memberOf" ,"CN","OU","name"});
   
   
   /* 
    This is the condition or filter that will be applied to pull results.
    This is more like where clause in your SQL statement.
    BEWARE: If you give more generic  search criteria then you will have too many records and may get errors.
    Its good practice to restrict or narrow down results to smaller subsets.
   */
   
   results = ctx.search("", "(objectclass=person)", controls);
   
   System.out.println(results);
   while (results.hasMore()) {
    SearchResult searchResult = (SearchResult) results.next();
    Attributes attributes = searchResult.getAttributes();
    //Ok lets pick few that we are interested.
    Attribute memeberOfAttr = attributes.get("memberOf");
    Attribute cnAttr = attributes.get("CN");
    Attribute nameAttr = attributes.get("name");
    Attribute samaAccountNameAttr = attributes.get("sAMAccountName");
    list.add(cnAttr.toString());
    
    
    //Debug only ...
    dumpAll(attributes);
    
    //memberOf in my case can be 1 to many .... So just show hot to pull if we have a collection of 
    //same attribute in LDAP.     
    if(memeberOfAttr!=null){
     String memeberOfStr = "";
     try{
      memeberOfStr = getAllValues(memeberOfAttr);
     }catch(Exception exp){
      exp.printStackTrace();
     }
     System.out.println("samaAccountName="+ samaAccountNameAttr+"| cnAttr="+ cnAttr  +" | name="+nameAttr +" | memeberof="+ memeberOfStr);
    }
    
   }
  } catch (NameNotFoundException e ) {
   // The base context was not found.
   // Just clean up and exit.
  } catch (NamingException e) {
   throw new RuntimeException(e);
  } finally {
   if (results != null) {
    try {
     results.close();
    } catch (Exception e) {
     // Never mind this.
    }
   }
   if (ctx != null) {
    try {
     ctx.close();
    } catch (Exception e) {
     // Never mind this.
    }
   }
  }
  return list;
 }


 private void dumpAll(Attributes attributes) throws NamingException {
  NamingEnumeration<? extends Attribute>  allAttr= attributes.getAll();
  while (allAttr.hasMoreElements()) {
   Attribute attribute = (Attribute) allAttr.nextElement();
   if((attribute.getAll()!= null) &&(attribute.getAll().hasMoreElements())){
    //Ok this Attributes can have one or more values.
    NamingEnumeration<?>  childAttr= attribute.getAll();
    while(childAttr.hasMoreElements()){
     System.out.println("Property="+ attribute.getID() +" Value="+ childAttr.next());
    }
   }else{
    System.out.println("Not All="+ attribute.get().toString());
   }
  }
 }
 
 public List<LDAPUser> getAllPersonNameObjs(){
  // I will show the demo later.
  return null;
 }
 
 
 /*
   Here memeberOf can have many rows.
   memberOf: CN=SCHOOL_MAINT_FULL,OU=Security_Groups_Distribution_Lists,DC=DevS1,DC=int
   memberOf: CN=REPORT_FULL,OU=Security_Groups_Distribution_Lists,DC=DevS1,DC=int
   memberOf: CN=LOAN_APP_FULL,OU=Security_Groups_Distribution_Lists,DC=DevS1,DC=int
   memberOf: CN=DISB_MAINT_READ,OU=Security_Groups_Distribution_Lists,DC=DevS1,DC=int
  */
 
 public static String getAllValues(Attribute attribute)throws Exception {
  StringBuffer sbuf = new StringBuffer();
  NamingEnumeration<?>  childAttr= attribute.getAll();
  while(childAttr.hasMoreElements()){
   Object obj = childAttr.next();
   sbuf.append("id="+ attribute.getID()+" Value="+ obj.toString()+"\n");
  }
  
  return sbuf.toString();
 }
 
 public static void main(String[] args) {
  LDAPUserRepo personRepo = new TraditionalLDAPUserRepoImpl();
  personRepo.getAllPersonNames();
 }
}

 


Exceptions:.
===========
Some times you will see exceptions like below which indicates that your search query is returning more results and LDAP server may have size restrictions.
The best thing to do is to change you were clause to limit the result set. I have notice that Spring LDAP template was able to fetch more records than plain java solution.

Exception in thread "main" java.lang.RuntimeException: javax.naming.SizeLimitExceededException: [LDAP: error code 4 - Sizelimit Exceeded]; remaining name ''at com.rama.util.activedirectory.TraditionalLDAPUserRepoImpl.getAllPersonNames(TraditionalLDAPUserRepoImpl.java:87)
at com.rama.util.activedirectory.TraditionalLDAPUserRepoImpl.main(TraditionalLDAPUserRepoImpl.java:151)
Caused by: javax.naming.SizeLimitExceededException: [LDAP: error code 4 - Sizelimit Exceeded]; remaining name ''

NOTE:
=====
For troubleshooting and also to just to view I use LDAP browser from Softterra (Google it). Its FREE. (Don't use "LDAP administrator" from softteraa thats a paid one)
Here is snapshot for my user id when I look it up through LDAP browser.
(Here is the link at the time I was writing this blog http://www.ldapadministrator.com/softerra-ldap-browser.htm)











Now the same thing can be done using Spring LdapTemplate.
---------------------------------------------------------
The beauty of Spring ldap template is you can filter data using AndFilter , OrFilter , LikeFilter.
Please see spring package org.springframework.ldap.filter.* for all filters.
The below code will show you how to use filters.

Ok lets create the maven java project.Here is the maven pom.xml.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>

 <groupId>com.rama.activedirectory</groupId>
 <artifactId>FMActiveDirectory</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>jar</packaging>
 <name>FMActiveDirectory</name>
 <url>http://maven.apache.org</url>

 <properties>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
 </properties>

 <dependencies>
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <version>3.8.1</version>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupId>org.springframework.ldap</groupId>
   <artifactId>spring-ldap-core</artifactId>
   <version>2.0.4.RELEASE</version>
  </dependency>
  <dependency>
   <groupId>org.springframework.data</groupId>
   <artifactId>spring-data-commons-core</artifactId>
   <version>1.4.1.RELEASE</version>
  </dependency>
  <dependency>
   <groupId>commons-lang</groupId>
   <artifactId>commons-lang</artifactId>
   <version>2.6</version>
  </dependency>
 </dependencies>
</project>

Now we need to return as java object so lets create a POJO to get the values.

I will add few properties feel free to enhance it.


package com.rama.util.activedirectory;
import java.util.List;

public class LDAPUser {
 private String sAMAccountName ;
 private String name;
 private String givenName;
 private String canonicalName;
 private List<String> memberOfValueList  = new ArrayList<String>();
 
 // Removed getter and setters for clarity, Please generate it.
 
}

Spring configuration.
===================
Here is how the spring xml looks. I called it spring-config.xml


<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:ldap="http://www.springframework.org/schema/ldap"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/ldap http://www.springframework.org/schema/ldap/spring-ldap.xsd">

   <ldap:context-source
       id="ldapcontextSource" 
          url="ldap://dev.rama.int:389/DC=DevS1,DC=int"  base="DC=DevS1,DC=int"
          username="CN=rama,OU=Service_Account,DC=DevS1,DC=int"  password="test123" />

 <ldap:ldap-template id="ldapTemplate"  context-source-ref="ldapcontextSource"/>

   <bean id="ldapUserRepoBean" class="com.rama.util.activedirectory.SpringTemplateLDAPUSerRepoImpl">
      <property name="ldapTemplate" ref="ldapTemplate" />
   </bean>
</beans> 
Make sure this spring xml file is in your classpath.
Now we need to implement our LDAPUserRepo interface.In order to convert the result set back to java object we need to also implement AttributesMapper interface. Please see the code below.

The code several filters that you can use like AndFilter , OrFilter, LikeFilter etc,
See package "org.springframework.ldap.filter.*" for more filter Options.

SpringTemplateLDAPUSerRepoImpl.java
--------------------------------

package com.rama.util.activedirectory;
import static org.springframework.ldap.query.LdapQueryBuilder.query;

import java.util.ArrayList;
import java.util.List;

import javax.naming.NamingEnumeration;
import javax.naming.NamingException;
import javax.naming.directory.Attribute;
import javax.naming.directory.Attributes;

import org.springframework.beans.factory.BeanFactory;
import org.springframework.beans.factory.xml.XmlBeanFactory;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.Resource;
import org.springframework.ldap.core.AttributesMapper;
import org.springframework.ldap.core.LdapTemplate;
import org.springframework.ldap.filter.OrFilter;
import org.springframework.ldap.query.LdapQuery;

public class SpringTemplateLDAPUSerRepoImpl implements LDAPUserRepo  {

  private LdapTemplate ldapTemplate;

  public void setLdapTemplate(LdapTemplate ldapTemplate) {
      this.ldapTemplate = ldapTemplate;
   }
    
 /**
  * No Attribute mapper. Simple pull of each property.
  * 
  * @return
  */
 public List<LDAPUser> getAllPersonNameObjs(){
  
  
  //NOTE: Spring doesn't throw javax.naming.SizeLimitExceededException:. Though it cant fetch 
  // all records its remains siletnt.
  // OOPS this is too broad. It may get all memebers from LDAP. But there is a limitation of 1000 in 
  // my case and Spring ldap template pulls thousand and keeps Quite. No exceptions 
  
  LdapQuery query = query().where("objectclass").is("person");
  ldapTemplate.setIgnorePartialResultException(true);
  return ldapTemplate.search(query,new LDAPUSerAttributesMapper());
  
  
  //TO Return reduced result set I tried this to search based on one of the groups the user belongs to. 
  //LdapQuery query = query().where("memberOf").is("CN=DEV_ADMIN_FULL,OU=Security_Groups_Distribution_Lists,DC=DevS1,DC=int");
  //ldapTemplate.setIgnorePartialResultException(true);
  //return ldapTemplate.search(query,new LDAPUSerAttributesMapper());
  
  
  
  //AndFilter andFilter = new AndFilter();
      //andFilter.and(new EqualsFilter("objectclass","person"));
      //andFilter.and(new EqualsFilter("cn",firstName));
      //andFilter.and(new EqualsFilter("sn",lastName));
      //andFilter.and(new EqualsFilter("ou","User_Accounts"));
   //System.out.println("LDAP Query " + filter.encode());
      //ldapTemplate.setIgnoreSizeLimitExceededException(true);
      //ldapTemplate.setIgnorePartialResultException(true);
         //return ldapTemplate.search("",filter.encode(),new LDAPUSerAttributesMapper());
      
   
  //OrFilter filter = new OrFilter();
    //filter.or(new EqualsFilter("objectclass","person"));
    //filter.or(new EqualsFilter("name","Consultants_Contractors_Temps"));
    //distinguishedName: CN=Ramachandra Reddy,OU=Consultants_Contractors_Temps,OU=User_Accounts,DC=DevS1,DC=int
    //filter.or(new EqualsFilter("distinguishedName","CN=Ramachandra Reddy,OU=Consultants_Contractors_Temps,OU=User_Accounts,DC=DevS1,DC=int"));
   //System.out.println("LDAP Query " + filter.encode());
      //ldapTemplate.setIgnoreSizeLimitExceededException(true);
      //ldapTemplate.setIgnorePartialResultException(true);
         //return ldapTemplate.search("",filter.encode(),new LDAPUSerAttributesMapper());
      
   
  //LikeFilter filter = new LikeFilter("memberOf","CN=DEVS1_DEV_Inhouse_SecGroup,OU=Security_Groups_Distribution_Lists,DC=DevS1,DC=int");
   //OrFilter filter = new OrFilter();
   //filter.or(new EqualsFilter("memberOf","CN=DEVS1_DEV_Inhouse_SecGroup,OU=Security_Groups_Distribution_Lists,DC=DevS1,DC=int"));
   //filter.or(new EqualsFilter("memberOf","CN=LOS_ADMIN_FULL,OU=Security_Groups_Distribution_Lists,DC=DevS1,DC=int"));
      //System.out.println("LDAP Query " + filter.encode());
      //ldapTemplate.setIgnoreSizeLimitExceededException(true);
      //ldapTemplate.setIgnorePartialResultException(true);
         //return ldapTemplate.search("",filter.encode(),new LDAPUSerAttributesMapper());
      
   
 }
 
 
 public List<String> getAllPersonNames() {
       return ldapTemplate.search(
          query().where("objectclass").is("user"),
          new AttributesMapper<String>() {
             public String mapFromAttributes(Attributes attrs)
                throws NamingException {
                return (String) attrs.get("cn").get();
             }
          });
   }
 
 /*
  This is how you map your results back to a 
  Java object
 
 */
  private class LDAPUSerAttributesMapper implements AttributesMapper<LDAPUser> {
       public LDAPUser mapFromAttributes(Attributes attrs) throws NamingException {
       LDAPUser person = new LDAPUser();
          person.setsAMAccountName(getValue(attrs.get("sAMAccountName")));
          person.setName(getValue(attrs.get("name")));
          person.setGivenName(getValue(attrs.get("givenName")));
          person.setCanonicalName(getValue(attrs.get("canonicalName")));
          try {
       //Special handling to pull this property as it can  be one to many.
    person.setMemberOfValueList(getAllValues(attrs.get("memberOf")));
   } catch (Exception e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
   }
          return person;
       }
    }
  
  /**
   * Checks for null and accordingly gets String value of the attribute.
  * @param attr
  * @return
  * @throws NamingException
  */
 private static String getValue(Attribute attr)throws NamingException{
   if((attr!= null) &&(attr.get() != null)){
     return (String)attr.get();
   }
    return "";
  }
 
 private static void dumpAll(Attributes attributes) throws NamingException {
  NamingEnumeration<? extends Attribute>  allAttr= attributes.getAll();
  while (allAttr.hasMoreElements()) {
   Attribute attribute = (Attribute) allAttr.nextElement();
   System.out.println("All="+ attribute.get().toString());
  }
  
 }
 
 
 public static List<String> getAllValues(Attribute attribute)throws Exception {
  List<String> valueList = new ArrayList<String>();
  if(attribute != null){
   NamingEnumeration<?>  childAttr= attribute.getAll();
   while(childAttr.hasMoreElements()){
    Object obj = childAttr.next();
    //System.out.println("id="+ attribute.getID()+" Value="+ obj.toString()+"\n");
    valueList.add(obj.toString());
   }
  }
  return valueList;
 }
 
 public static void main(String[] args) {
          Resource resource = new ClassPathResource("/spring-config.xml");
          BeanFactory factory = new XmlBeanFactory(resource);
          LDAPUserRepo ref = (LDAPUserRepo) factory.getBean("ldapUserRepoBean");
          System.out.println(ref.getAllPersonNameObjs());
          System.out.println("Done.");
 }
}

As shown the ldaptemplate is much easier and has more flexibility so that we can use many filters, nest the filters to restrict data from  LDAP active directory. So needless to say Spring approach is much easier.