You are viewing kyrill007

Previous Entry | Next Entry


JSON serialization format is all the rage these days when it comes to making Ajax calls from the browser. And Jackson JSON-processor is probably the best and most popular serializer written in Java for converting domain objects into JSON. Hibernate is, of course, the most popular object-relational mapping framework. And the problem is that those three don’t play well together...

The main obstacle is lazily initialized object properties: whenever a graph of object is retrieved from Hibernate, it is necessary to limit the amount of data, thus some properties must be lazy-loaded. And this is where the things begin to break down. There is really no easy workaround: if the session remains open during JSON serialization, then Jackson is going to walk through your object graph and lazily instantiate every object, potentially loading the whole database in memory; if the session is closed, then the moment a "lazy" property is encountered the org.hibernate.LazyInitializationException is going to be thrown.

I was really surprised to discover that there has been really no acceptable solution to the problem yet. Some folks advocate building a DTO layer for doing the conversion in code, some suggest inspecting JPA/Hibernate annotations on entities and reject any property marked as FetchType.LAZY, which effectively kills object graph navigation as FetchType.LAZY is the dominant form of connecting entities together. The issue is well documented in Jackson JIRA Item 276.

So, I decided to dig into Jackson internals and come up with a solution. The main requirement was that the solution must not interfere with the domain model and be transparent. It should also satisfy the following criteria:

1.       Must transparently detect if a property is “lazy” and serialize it as “null”.

2.       Must not depend only on annotation mappings. If an entity is mapped in XML, that should be fine too.

3.       Must support byte-code instrumented Hibernate entities, i.e. the ones that allow making simple fields lazy (not just entity properties or collections).  

I’ve accomplished this by creating a custom Jackson SerializerFactory: HibernateAwareSerializerFactory. Here is its source code. I've put as many comments as I could, so it should be self-documenting.

import javax.persistence.Transient;
import java.beans.PropertyDescriptor;
import java.io.IOException;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;

import org.codehaus.jackson.JsonGenerator;
import org.codehaus.jackson.map.JsonSerializer;
import org.codehaus.jackson.map.SerializationConfig;
import org.codehaus.jackson.map.SerializerProvider;
import org.codehaus.jackson.map.introspect.BasicBeanDescription;
import org.codehaus.jackson.map.ser.BeanPropertyWriter;
import org.codehaus.jackson.map.ser.BeanSerializerFactory;
import org.codehaus.jackson.type.JavaType;
import org.hibernate.bytecode.javassist.FieldHandled;
import org.hibernate.collection.PersistentCollection;
import org.hibernate.collection.PersistentMap;
import org.hibernate.proxy.HibernateProxy;
import org.springframework.beans.BeanUtils;
import org.springframework.core.annotation.AnnotationUtils;

/**
 * This is the key class in enabling graceful handling of Hibernate managed entities when
 * serializing them to JSON.
 * <p/>
 * The key features are:
 * 1) Non-initialized properties will be rendered as {@code null} in JSON to prevent
 * "lazy-loaded" exceptions when the Hibernate session is closed.
 * 2) {@link Transient} properties not be rendered at all as they often present back door
 * references to non-initialized properties.
 *
 * @author Kyrill Alyoshin
 */
public class HibernateAwareSerializerFactory extends BeanSerializerFactory {
    /**
     * Name of the property added during build-time byte-code instrumentation
     * by Hibernate. It must be filtered out.
     */
    private static final String FIELD_HANDLER_PROPERTY_NAME = "fieldHandler";

    @Override
    @SuppressWarnings("unchecked")
    public JsonSerializer<Object> createSerializer(JavaType type, SerializationConfig config) {
        Class<?> clazz = type.getRawClass();

        //check for all Hibernate proxy invariants and build custom serializers for them
        if (PersistentCollection.class.isAssignableFrom(clazz)) {
            return new PersistentCollectionSerializer(type, config);
        }

        if (HibernateProxy.class.isAssignableFrom(clazz)) {
            return new HibernateProxySerializer(type, config);
        }

        //Well, then it is not a Hibernate proxy
        return super.createSerializer(type, config);
    }

    /**
     * The purpose of this method is to filter out {@link Transient} properties of the bean
     * from JSON rendering.
     */
    @Override
    protected List<BeanPropertyWriter> filterBeanProperties(SerializationConfig config,
                                                            BasicBeanDescription beanDesc,
                                                            List<BeanPropertyWriter> props) {

        //filter out standard properties (e.g. those marked with @JsonIgnore)
        props = super.filterBeanProperties(config, beanDesc, props);

        filterInstrumentedBeanProperties(beanDesc, props);

        //now filter out the @Transient ones as they may trigger "lazy" exceptions by
        //referencing non-initialized properties
        List<String> transientOnes = new ArrayList<String>();
        //BeanUtils and AnnotationUtils are utility methods that come from
        //the Spring Framework
        for (PropertyDescriptor pd : BeanUtils.getPropertyDescriptors(beanDesc.getBeanClass())) {
            Method getter = pd.getReadMethod();
            if (getter != null && AnnotationUtils.findAnnotation(getter, Transient.class) != null) {
                transientOnes.add(pd.getName());
            }
        }

        //remove transient
        for (Iterator<BeanPropertyWriter> iter = props.iterator(); iter.hasNext();) {
            if (transientOnes.contains(iter.next().getName())) {
                iter.remove();
            }
        }

        return props;
    }

    private void filterInstrumentedBeanProperties(BasicBeanDescription beanDesc,
                                                  List<BeanPropertyWriter> props) {

        //all beans that have build-time instrumented lazy-loaded properties
        //will implement FieldHandled interface.
        if (!FieldHandled.class.isAssignableFrom(beanDesc.getBeanClass())) {
            return;
        }

        //remove fieldHandler bean property from JSON serialization as it causes
        //infinite recursion
        for (Iterator<BeanPropertyWriter> iter = props.iterator(); iter.hasNext();) {
            if (iter.next().getName().equals(FIELD_HANDLER_PROPERTY_NAME)) {
                iter.remove();
            }
        }
    }

    /**
     * The purpose of this class is to perform graceful handling of custom Hibernate collections.
     */
    private class PersistentCollectionSerializer extends JsonSerializer<Object> {
        private final JavaType type;
        private final SerializationConfig config;

        private PersistentCollectionSerializer(JavaType type, SerializationConfig config) {
            this.type = type;
            this.config = config;
        }


        @Override
        @SuppressWarnings("unchecked")
        public void serialize(Object value, JsonGenerator jgen, SerializerProvider provider) throws IOException {
            //avoid lazy initialization exceptions
            if (!((PersistentCollection) value).wasInitialized()) {
                jgen.writeNull();
                return;
            }

            //construct an actual serializer from the built-in ones
            BasicBeanDescription beanDesc = config.introspect(type.getRawClass());
            Class<?> clazz = type.getRawClass();

            JsonSerializer<Object> serializer;
            if (PersistentMap.class.isAssignableFrom(clazz)) {
                serializer = (JsonSerializer<Object>) buildMapSerializer(type, config, beanDesc);
            }
            else {
                serializer = (JsonSerializer<Object>) buildCollectionSerializer(type, config, beanDesc);
            }

            //delegate serialization to a built-in serializer
            serializer.serialize(value, jgen, provider);
        }
    }

    /**
     * The purpose of this class is to perform graceful handling of HibernateProxy objects.
     */
    private class HibernateProxySerializer extends JsonSerializer<Object> {
        private final JavaType type;
        private final SerializationConfig config;

        private HibernateProxySerializer(JavaType type, SerializationConfig config) {
            this.type = type;
            this.config = config;
        }

        @Override
        public void serialize(Object value, JsonGenerator jgen, SerializerProvider provider) throws IOException {
            if (((HibernateProxy) value).getHibernateLazyInitializer().isUninitialized()) {
                jgen.writeNull();
                return;
            }

            BasicBeanDescription beanDesc = config.introspect(type.getRawClass());
            JsonSerializer<Object> serializer = findBeanSerializer(type, config, beanDesc);

            //delegate serialization to a build-in serializer
            serializer.serialize(value, jgen, provider);
        }
    }
}


And this point this custom SerializerFactory needs to be registered with the root Jackson ObjectMapper. If you're using IoC container (like Spring) to wire up your project infrastructure, it is best to create your own ObjectMapper to tweak it a bit:

import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.map.SerializationConfig.Feature;

/**
 * This class extends {@code ObjectMapper} class of the Jackson framework to provide
 * minor customizations:
 * <ul>
 * <li>To set a custom {@link HibernateAwareSerializerFactory}</li>
 * <li>To relax Jackson handling of unknown class types</li>
 * </ul>
 * <p/>
 * <em>Note:</em> Due to the nature {@code ObjectMapper} class
 * those customization could not be done through the Spring Framework.
 *
 * @author Kyrill Alyoshin
 * @see HibernateAwareSerializerFactory
 */
public class HibernateAwareObjectMapper extends ObjectMapper {

    public HibernateAwareObjectMapper() {
        setSerializerFactory(new HibernateAwareSerializerFactory());
        configure(Feature.FAIL_ON_EMPTY_BEANS, false);
    }

    public void setPrettyPrint(boolean prettyPrint) {
        configure(Feature.INDENT_OUTPUT, prettyPrint);
    }
}


And this is it.

I do have a comprehensive integration testing suite in my project to test these classes.

The only issue that still needs to be address is bi-directional navigation. Jackson will run out of stack when it attempts to serialize bi-directional entities. So, it is important to mark one side of the association with @JsonIgnore annotation. After this is done, you should be able to serialize your “nurtured” domain model into JSON using Jackson without resorting to useless DTO layers or one-off solutions.

Comments

( 33 comments — Leave a comment )
(Anonymous)
Oct. 6th, 2010 02:12 pm (UTC)
having some problems
Hi,

I tried your approach handling.

First the the class HibernateAwareSerializerFactory won't compile for me. I had either to change the two line containing config.introspect(type.getRawClass()); to config.introspectClassAnnotations(type.getRawClass()); or to config.introspect(type);

Then the class get's compiled without any errors. I tried to use the Mapper in my spring app using:






However I'm still running in:
org.codehaus.jackson.map.JsonMappingException: failed to lazily initialize a collection of role ...

Any ideas?

regards,
Frank
kyrill007
Oct. 6th, 2010 02:27 pm (UTC)
Re: having some problems
Right, I forgot to mention that the code was tested on 1.5.x version of Jackson. It looks like he changed the 'SerializationConfig#introspect' method signature in 1.6.0. The code obviously goes pretty deeply into Jackson internals, so I am not surprised that something may have changed internally there to cause lazy initialization exceptions. Give it a shot on 1.5.x and see if it works... It should. If not, I'd like to see your Hibernate mapping.
(Anonymous)
Oct. 6th, 2010 02:48 pm (UTC)
Re: having some problems
Hi,

wow that was fast :-). Yes you are right, using the 1.5.7er version it compiles without any failures. However I'm getting the same failure concerning lazy loading. The HibernateAwareObjectMapper constructor get's called during startup, but not the HibernateAwareSerializerFactory during a request. You can find my spring config here: http://pastebin.com/k8n86ABe.

thanks for you help,
Frank
kyrill007
Oct. 6th, 2010 02:58 pm (UTC)
Re: having some problems
I am concerned there with multiple MappingJacksonHttpMessageConverter's in your Spring config. One has HibernateAwareObjectMapper wired in, the other one does not. It does seem like a Spring misconfiguration. I looked at the Spring's source code for MappingJacksonHttpMessageConverter, and it looks solid; once correct object mapper is supplied, everything should work.
(Anonymous)
Oct. 6th, 2010 03:23 pm (UTC)
Re: having some problems
Hi,

I'm sorry still having the same problem with lazy initialization:
org.codehaus.jackson.map.JsonMappingException: failed to lazily initialize a collection of role: com.loiane.model.KFCase.KFCaseSections, no session or session was closed

I copied my spring config again ... http://pastebin.com/dexHzYkk.

Any more ideas?

regards,
Frank
kyrill007
Oct. 6th, 2010 03:30 pm (UTC)
Re: having some problems
Use



as a view within your InternalResourceViewResolver.

It is my understanding that MappingJacksonHttpMessageConverter is for RESTful WS calls. If you just want to get a web app going with some JSON spitting back-end, then MappingJacksonJsonView is the way to go.
kyrill007
Oct. 6th, 2010 03:32 pm (UTC)
Re: having some problems
XML didn't copy paste.

bean class="org.springframework.web.servlet.view.json.MappingJacksonJsonView"
property name="objectMapper" ref="hibernateAwareObjectMapper"
bean
(Anonymous)
Oct. 6th, 2010 04:38 pm (UTC)
Re: having some problems
You are my hero of the day... Thanks a lot for your time and effort. Hopefully this gets integrated in some way.

regards,
Frank
kyrill007
Oct. 6th, 2010 05:25 pm (UTC)
Re: having some problems
Glad it worked out!
kyrill007
Oct. 6th, 2010 02:35 pm (UTC)
Re: having some problems
So, I just plugged in 1.6.0. You do need to change config.introspect(type.getRawClass()) to config.introspect(type). Other than that all of my integration tests passed with flying colors. I definitely would like to see the mapping of your Hibernate entities.
(Anonymous)
Oct. 6th, 2010 02:57 pm (UTC)
Re: having some problems
Hi,

I added the beans and db config ... http://pastebin.com/MN5mPi0C.
I'm pretty new to spring mvc :-)

regards,
Frank
(Anonymous)
Nov. 10th, 2010 06:05 pm (UTC)
Thanks
Thanks, you saved our day :-)
(Anonymous)
Feb. 15th, 2011 08:01 pm (UTC)
Jackson-module-hibernate
Quick note: there is now new Jackson module at: https://github.com/FasterXML/jackson-module-hibernate which is built using this article's ideas and Kirill's help.
(Anonymous)
Mar. 9th, 2011 03:11 am (UTC)
Review Bebbled Review
At this point, there is very little to recommend Gmail Backup over Gmail Keeper, except this: Gmail Backup is free. If basic functionality is all you need, there's no sense in paying for features you won't use; Gmail Keeper's $30 price tag is well out of "impulse buy" range for a narrowly focused utility. Gmail Backup's open-source nature may be a valuable plus for some readers. If development does begin to pick up as promised, it may well add in more functionality over time.
(Anonymous)
Mar. 31st, 2011 07:21 am (UTC)
Exclude those getMethods whose Field is marked Transient with transient java keyword
Hi,

I have modified filterBeanProperties to exclude those getMethods whose Field is marked Transient with Transient java keyword.


/**
* The purpose of this method is to filter out {@link Transient} properties of the bean
* from JSON rendering.
*/
@Override
protected List filterBeanProperties(SerializationConfig config,
BasicBeanDescription beanDesc,
List props) {

//filter out standard properties (e.g. those marked with @JsonIgnore)
props = super.filterBeanProperties(config, beanDesc, props);

filterInstrumentedBeanProperties(beanDesc, props);

//now filter out the @Transient ones as they may trigger "lazy" exceptions by
//referencing non-initialized properties
List transientOnes = new ArrayList();
//BeanUtils and AnnotationUtils are utility methods that come from
//the Spring Framework
for (PropertyDescriptor pd : BeanUtils.getPropertyDescriptors(beanDesc.getBeanClass())) {

try {

Field fld = beanDesc.getBeanClass().getDeclaredField(pd.getName());
if( Modifier.isTransient( fld.getModifiers() )){

Method getter = pd.getReadMethod();
if ( getter != null ) {
transientOnes.add(pd.getName());
}
continue;
}
} catch (NoSuchFieldException e) {

//Ignore it as some reflected field may not be available
//e.printStackTrace();
}
catch (Exception e) {
//TODO
e.printStackTrace();
}

Method getter = pd.getReadMethod();
if (getter != null && AnnotationUtils.findAnnotation(getter, Transient.class) != null) {
transientOnes.add(pd.getName());
}
}

//remove transient
for (Iterator iter = props.iterator(); iter.hasNext();) {
if (transientOnes.contains(iter.next().getName())) {
iter.remove();
}
}

return props;
}


Regards,
Rashid
Eran Medan
May. 11th, 2011 05:30 am (UTC)
Works like a charm, but can you elaborate how do I go about limiting the tree graph?
This works like a charm! however, it does get me quite big graphs :)

Is there a way to make it work only with already initialized entities and not attempt to initialize anything (or am I wrong and it doesn't)? I'd like the service to decide what to initialize and when
Can it be done? (or did I miss something and you already do that) :)

Thanks again
kyrill007
May. 11th, 2011 01:19 pm (UTC)
Re: Works like a charm, but can you elaborate how do I go about limiting the tree graph?
Yes, you missed it all. :-) This object mapper does not load uninitialized Hibernate proxies but stops at them and converts them to JavaScript 'nulls'. So, if you object graph is too big, it is because that's what you're supplying to the mapper. :-)
kyrill007
May. 11th, 2011 01:21 pm (UTC)
Re: Works like a charm, but can you elaborate how do I go about limiting the tree graph?
And, of course, I was talking about my code, i.e. the content of the actual post, not the comments with modifications.
Eran Medan
May. 19th, 2011 01:01 am (UTC)
Re: Works like a charm, but can you elaborate how do I go about limiting the tree graph?
Yes, a minute after I posted I looked again at the code and realized this is just what you do :) and indeed it works like a charm!

My problem now is with Generics, and I seen somewhere that Jackson 1.8 should take care of it, so I downloaded it but now it broke your code :) it doesn't compile... I tried to do some quick fixes here, but besides passing nulls in new introduced parameters to buildCollectionSerializer and buildMapSerializer I have no clue what to do

Is there a chance you have a 1.8 compatible version by any chance? ;)
kyrill007
May. 19th, 2011 01:06 am (UTC)
Re: Works like a charm, but can you elaborate how do I go about limiting the tree graph?
Eran Medan
May. 19th, 2011 03:59 am (UTC)
Re: Works like a charm, but can you elaborate how do I go about limiting the tree graph?
Thank you! :)
(Anonymous)
Jun. 2nd, 2011 02:26 pm (UTC)
Deserializing objects
Hi, this is good stuff thanks for the post. I'm curious if you have tackled turning json into hibernate pojos.
kyrill007
Jun. 2nd, 2011 02:28 pm (UTC)
Re: Deserializing objects
Hibernate pojos are just Java beans. Jackson should be able to transparently populate those. What is the actual issue?
(Anonymous)
Jun. 10th, 2011 08:10 am (UTC)
Infinite recursion problem while fetching the newly saved record
Hi Kyrill. Thanks for posting the solution. We are using spring(3.0.3.RELEASE) + JPA(spring-jap2.0.8) + Hibernate(3.5.4-Final) + Jackson(1.8.1) + Jersey(1.7) in our project.
The lazy load problem while using "mapper.writeValueAsString()" was solved using the above solution.

But the problem we are facing right now is like this -
We use javax enetitymanager for making DB transactions.
We create an entity which has bi-directional associations (using em.persist(entity)). The record gets saved very nicely.
We load the newly saved record immediately, but we get infinite recursion problem :
at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:212)

What could be the way to avoid this problem ??
We understand this is due to bi-directional dependency and we don't want to use @JsonIgnore also.
kyrill007
Jun. 10th, 2011 12:31 pm (UTC)
Re: Infinite recursion problem while fetching the newly saved record
Yes, if you don't want to use @JsonIgnore on one of the ends of a bi-directional association, then you need to write your own JSON serialization framework that is more clever than Jackson. :-)
(Anonymous)
Jun. 12th, 2011 09:17 am (UTC)
Re: Infinite recursion problem while fetching the newly saved record
Thanks Kyrill. Actually, I tried using @JsonBackReference and @JsonManagedReference, and it solved the problem :))
But have another challenge now. The pojos are auto generated using hbm2java goal of hibernate-tools maven plugin with reverse engineering, and that won't generate these custom annotations (may be I am not aware of how to do this.) :(
(Anonymous)
Jun. 11th, 2011 09:44 pm (UTC)
u rock
Thanks a lot for the solution ..It worked great!
(Anonymous)
Jul. 18th, 2011 10:05 am (UTC)
Thank you! Super solution. Would like to provide an enhancement suggestion for future :-)!
Thanks Kyrill! This solution did work.

My other problem is that the model POJO's are auto-generated via HBM files as it has been reported by the other user above. So are there any plans in future to make this solution work probably by providing some exclusion rules explicitly on the objectmapper ? instead of the @JsonIgnore annotation? (I have seen some JSONGenerators providing this sort of exclusion rules, so just curious.)

However for now its not a very painful exercise as the list of HBMs are only handful and I can go and modify these POJOs myself. But can see this becoming a bottleneck when the list grows huge and each subsequent code generation could become painful.

Overall your solution is the best I have searched so far! Thank You!
(Anonymous)
Oct. 19th, 2011 03:49 am (UTC)
How do I wire this up
I am new to Spring and Json, I have added the 2 clean compiled classes to my project but I don'y know how to wire it all up.
(Anonymous)
Jun. 30th, 2012 12:14 pm (UTC)

Hi,

getting abuch of errors with current jackson mapper im using:

org.codehaus.jackson
jackson-mapper-asl
1.9.2



1. ERROR in /Developer/dev/tabber/tabber/dao/src/main/java/com/xx/ql/json/HibernateAwareSerializerFactory.java (at line 36)
public class HibernateAwareSerializerFactory extends BeanSerializerFactory {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Implicit super constructor BeanSerializerFactory() is undefined for default constructor. Must define an explicit constructor
----------
2. ERROR in /Developer/dev/tabber/tabber/dao/src/main/java/com/xxx/ql/json/HibernateAwareSerializerFactory.java (at line 45)
public JsonSerializer
(Anonymous)
Jun. 30th, 2012 12:15 pm (UTC)
Cannot override the final method from SerializerFactory

Hi,

getting abuch of errors with current jackson mapper im using:

org.codehaus.jackson
jackson-mapper-asl
1.9.2



1. ERROR in /Developer/dev/tabber/tabber/dao/src/main/java/com/xx/ql/json/HibernateAwareSerializerFactory.java (at line 36)
public class HibernateAwareSerializerFactory extends BeanSerializerFactory {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Implicit super constructor BeanSerializerFactory() is undefined for default constructor. Must define an explicit constructor
----------
2. ERROR in /Developer/dev/tabber/tabber/dao/src/main/java/com/xxx/ql/json/HibernateAwareSerializerFactory.java (at line 45)
public JsonSerializer
kyrill007
Jun. 30th, 2012 01:28 pm (UTC)
Re: Cannot override the final method from SerializerFactory
Yes, this code is based on the old version of Jackson. I believe this thread contains a link to a Jira issues that talks about how to resolve it for newer versions.
(Anonymous)
Oct. 2nd, 2012 09:01 pm (UTC)
Very Nice
Very Nice...thank you for the detailed explanation and solution.
( 33 comments — Leave a comment )