Ian Nicholas

Serializing Complex Python Data to JSON

Python has a JSON module that can serialize data structures like lists, tuples, and dictionaries to a JSON string. This comes in handy when you're serving up some content through AJAX. Unfortunately, your fancy Python data structure will sometimes contain objects that don't fit easily into JSON, and the parser won't even try to work with it. Instead, you'll get something nasty like this:

TypeError at /data/
Decimal('9000.00') is not JSON serializable

I get around this in Python 2.7 by writing a wrapper around json.dumps() that recurses through the data and performs type-specific cleanup.

import json

# Some data types we want to check for.
from datetime import date, datetime
from decimal import Decimal


def jsonify(data):

  # Get all that nasty Python cleaned up.
  data = jsonClean(data)

  # Now we should be safe to dump to a string.
  try:
    data = json.dumps(data)
  except TypeError:
    # Go home, data. You're drunk.
    data = 'Wut'

  return data


def jsonClean(data):

  # First things first: we need to do some simple cleanup of built-in 
  # Python types.

  # Sets. JSON doesn't know how to enforce uniqueness, but since we're 
  # probably done adding items here anyway, we should be safe turning 
  # it into a list.
  if isinstance(data, set):
    data = list(data)

  # Lists and tuples themselves translate into JSON arrays just fine, but 
  # the data they contain may not, so we have to recurse through it.
  if isinstance(data, list) or isinstance(data, tuple):
    # Make sure the data is mutable before we try making changes.
    data = list(data)
    for i in xrange(len(data)):
      data[i] = jsonClean(data[i])
    return data

  # Same thing with dictionary values.
  if isinstance(data, dict):
    for k, v in data.items():
      data[k] = jsonClean(v)
    return data

  # OK, now that we're beyond the sequences and dictionaries, we can get 
  # into formatting special cases. We'll pair data types with cleaning 
  # functions here.
  typeRules = {
    Decimal: cleanDecimal,
    date: cleanDate,
    datetime: cleanDate,
  }

  # And run the data through the appropriate cleaner.
  for typeName, typeCleaner in typeRules.items():
    if isinstance(data, typeName):
      return typeCleaner(data)

  # At this point we don't know what sort of data this is. Return it 
  # as-is and hope for the best.
  return data


# Turn a good precise decimal into a more JavaScript-friendly float.
def cleanDecimal(data):
  return float(data)


# Use an isoformat string for dates and times.
def cleanDate(data):
  return data.isoformat()

Then you can just import the jsonify function above and pass in any sort of data.

>>> data = {
...   'time': datetime.now(),
...   'numbers': set([Decimal('867.5309'), Decimal('42')]),
... }
>>> jsonify(data)
'{"numbers": [867.5309, 42.0], "time": "2013-08-10T02:17:20.572302"}'
>>>

This may not work for you out of the box because your data could contain other un-serializable data types that I didn't account for. There sure are a lot of them even if you don't venture into the third-party modules. But hopefully this framework will make it easy to hammer your data into shape.