Peter Hoffmann

Neo4j Podcasts

2012-05-01T00:00:00Z

The IBM Java technology zone technical podcast series has two nice podcasts about the Neo4j Graph Database:

Emil Eifrem, the man behind Neo4j (mp3)
Agile architect Peter Bell on Neo4j, a graph-oriented datastore built in Java (mp3)

The whole podcast series is worth listening, so subscribe to the RSS Feed.

Google Developer Weekend Hack and Tell

2011-11-18T00:00:00Z

The Google Developer Weekend started with a Berlin Hack and Tell Special Google Edition at C-Base.

Thanks to Deborah from http://9flats.com for the Pizza.

Here's a short list of the projects:

Friedger Muefke

http://open-accessories.com

Friedger showed how to modify google docs with google appscript and access data from a spreadsheet and send it to a blog.

Lean Engine

Mathias from lean engine showed how to cloud enable your mobile apps.

Stefan Wehrmeyer

Stefan presentedhttp://mapnificent.net, a layer over google maps which tells you what points in a city you can reach via public transport in a given time.

Stefan uses http://www.gtfs-data-exchange.com/ to get the timetables for public transport. Sadly the berlin transport organization BVG does not provide real time data.

wheelmap.org

whellmap.org is a map with wheelchair accessible places. The presenter showed how to work around the android sqlite limitations to work with lat/long positions.

Launchmode

Gonsala shode launchmode, an app to help visualize android activities.

xbounds.net

xbounds is a great hack to add screensharing to mobile devices.

andengine

Johannes presented how to use andengine together with Physics Editor. It's a simple way to add physics to your games (like collition detection). Just load an image of your level to Physics Editor and load the result to andengind.

komoot.de

Jonas and Jan talked about some design decisions for http://komoot.de an outdoor navigation and route planning app.

They dropped a local database on the mobile phones in support for a thin caching layer and now get all data from their servers.

Daniel Kurka

@dankurka showed m-gwt.com. It lets you programm gwt applications for mobile phones. The gwt programm is compiled to html5/css and runs via phone gap on your mobile phones

Barcamp Stuttgart 2011

2011-10-02T19:25:25Z

Barcamp Stuttgart 2011 is over and it was really nice, interesting and well organized (as always).

I chosed to participate in the following sessions:

Identity Management I visited the slot because I thought it would be about online identiy management, but the slot was about the corporate world. Tobias talked about different identity sources in corporations like SAP HR, Microsoft AD/Exchange and the problems on how to keep them up-to-date and in sync.
Memotechnik Andreas Lohrum introduced us to the art of memory and showed us how to remember the first 30 digits of PI. So now I only have to remember 100 words for the numbers 0 to 100 and than I should be able to remember telphone numbers. We'll see...
Startupscene Stuttgart What's the current status of the local startup scene and how to improve it. Harald Amelung made some Notes.
Geile Twitterbots Karsten Sauer showed us some of his twitter bots:
- @dblocator Who else is in your train.
- @einkaufsliste Manage your shopping list via twitter.
- @meineliste Manage any list via twitter.
#GO-NUTS Sven gave a short introduction into the go programming language

There could have been some more technical sessions. The quality of the speakers was very good.

send notes to simplenote from the command line and vim

2011-08-13T10:50:44Z

sn.py is a command line app to send text to simplenote service. It uses the simplenote.py library.:

#!/usr/bin/env python
"""simplenote command line app"""

import sys
import fileinput

from simplenote import Simplenote
sn = Simplenote("XXX", "XXX")


def send(txt):
    note = {'content': txt}
    note, status = sn.add_note(note)

    if status != 0:
        print >> sys.stderr, note
        sys.exit(1)


def read_until_dot():
    line = raw_input()
    while line != ".":
        yield line
        line = raw_input()

import sys
def main(argv=None):
    if argv is None:
        argv = sys.argv

    if len(argv) == 1:
        lines = read_until_dot()
    else:
        lines = fileinput.input()

    send("\n".join(lines))


if __name__ == '__main__':
    main()

Usage

Enter a note:

$ sn.py
write text
you want to send to simplenote
end note with one . in line
.

Pipe the output from another commant to simplenote:

$ echo "send output from command" | sn.py -

Send the contents from a file to simplenote:

$ sn.py some_text_file.txt

Vim

There is the simplenote.vim plugin. It lets you interact with simplenote. If you just want a quick way to send notes to simplenote use the following commands:

Send the current buffer to simplenote:

:%w !sn.py -

Send the current selection to simplenote:

:'<,'>w !sn.py -

Adding a dict based interface to the python leveldb api

2011-08-06T17:23:02Z

In the last post I showed how to install the python LevelDB bindings. Here is how to wrap the python LevelDB API into a dict like API with the help of UserDict.DictMixin. In a subclass we will extend the the simple key<string>:value<string> mapping to a dict with arbitrary JSON serializablie values.

from UserDict import DictMixin
import leveldb
import json


class LevelDict(DictMixin):
    """Dict Wrapper around the Google LevelDB Database"""
    def __init__(self, path):
        """Constructor for LevelDict"""
        self.path = path
        self.db = leveldb.LevelDB(self.path)

    def __getitem__(self, key):
        return self.db.Get(key)

    def __setitem__(self, key, value):
        self.db.Put(key, value)

    def __delitem__(self, key):
        self.db.Delete(key)

    def __iter__(self):
        for k, v in self.db.RangeIter():
            yield v

    def keys(self):
        return [k for k, v in self.db.RangeIter()]

    def iteritems(self):
        return self.db.RangeIter()

    def rangescan(self, start=None, end=None):
        if start is None and end is None:
            return self.db.RangeIter()
        elif end is None:
            return self.db.RangeIter(start)
        else:
            return self.db.RangeIter(start, end)

To store more than just strings we will add JSON serialization to a subclass:

class LevelJsonDict(LevelDict):
    """Dict Wrapper around the Google LevelDB Database with JSON serialization"""
    def __getitem__(self, key):
        return json.loads(LevelDict.__getitem__(self, key))

    def __setitem__(self, key, value):
        LevelDict.__setitem__(self, key, json.dumps(value))

    def iteritems(self):
        for k, v in LevelDict.iteritems(self):
            yield k, json.loads(v)

    def rangescan(self, start=None, end=None):
        for k, v in LevelDict.rangescan(self, start, end):
            yield k, json.loads(v)

Now you have a fast JSON Store with a Dict based Interface and a LevelDB backend.

Install LevelDB and the python bindings py-leveldb on ubuntu

2011-07-31T13:00:00Z

Google has open sourced LevelDB, a fast and lightweight key/value store. Chromium/Chrome uses LevelDB to power its IndexedD and there are bindings for Riak and Kyoto Tycoon (see Gabors Response on quora.com). The LevelDB Benchmark shows some impressive numbers compared against SQLite and Kyoto TreeDB.

Here are my installation notes on how to get LevelDB and the python bindings working on a fresh Ubuntu 11.04 installation.

Install development environment and get the LevelDB and py-leveldb source.

~$ sudo aptitude install build-essential python-dev
~$ mkdir ~/svn
~$ cd ~/svn
~/svn$ svn checkout http://py-leveldb.googlecode.com/svn/trunk/ py-leveldb-read-only
~/svn$ cd py-leveldb-read-only

After checking out the source I had to build leveldb by hand and than run the normal python setup.py install

~/svn/py-leveldb-read-only$ cd leveldb-read-only
~/svn/py-leveldb-read-only/leveldb-read-only$ make
~/svn/py-leveldb-read-only/leveldb-read-only$ cd ..
~/svn/py-leveldb-read-only$ sudo python setup.py install

Now you can fire up ipython and play around with the python api:

In [1]: import leveldb

In [2]: db = leveldb.LevelDB('/tmp/testdb')

In [3]: db.Put('some key', 'some val')

In [4]: db.Put('another key', 'another val')

In [5]: db.Get('some key')
Out[5]: 'some val'

In [6]: for a in db.RangeIter():
   ...:     print a
   ...:     
('another key', 'another val')
('some key', 'some val')

I'm going to do some micro benchmarks with the python bindings soon and add a UserDict based wrapper to the api. Until then there are some worth reading commments on the implementation and datastructures on news.ycombinator.

asq - a simple implementation of a LINQ-inspired API for python

2011-06-06T23:15:29Z

Robert Smallshire has released version 1.0 of asq:

asq is simple implementation of a LINQ-inspired API for Python which operates over Python iterables, including a parallel version implemented in terms of the Python standard library multiprocessing module.

An example what you can do with asq from the nice documentation

>>> students = [dict(firstname='Joe', lastname='Blogs', scores=[56, 23, 21, 89]),
            dict(firstname='John', lastname='Doe', scores=[34, 12, 92, 93]),
            dict(firstname='Jane', lastname='Doe', scores=[33, 94, 91, 13]),
            dict(firstname='Ola', lastname='Nordmann', scores=[98, 23, 98, 87]),
            dict(firstname='Kari', lastname='Nordmann', scores=[86, 37, 88, 87]),
            dict(firstname='Mario', lastname='Rossi', scores=[37, 95, 45, 18])]

>>> query(students).order_by(lambda s: query(s['scores']).average()) \
...                 .where(lambda student: student['firstname'].startswith('J')) \
...                 .select(lambda s: s['firstname'] + ' ' + s['lastname']) \
...                 .to_list()
['Joe Blogs', 'John Doe', 'Jane Doe']

You can do the same with list comprehension/generator expressions:

def avgscore(s):
	return avg(s['scores'])

[s['firstname']+' '+s['lastname'] for s in sorted(students, key=avgscore) if s['firstname'].startswith('J')]

But as discussed on reddit asq has some nice features like partial sorting and support for performing queries in parallel across multiple cores.

Authenticate Trac against a MySQL Table

2011-05-28T18:00:00Z

Recently I had to set up a trac instance which had to reuse authentification from an existing application.

Using Apache2 with Auth_MySQL it's really easy:

    ScriptAlias /trac /var/www-trac/deploy/cgi-bin/trac.fcgi/
    <Location "/trac">
            AuthName "Please Authenticate"
            AuthType Basic
            require valid-user
            AuthUserFile /dev/null
            AuthBasicAuthoritative Off

            Auth_MySQL On
            Auth_MySQL_Host localhost
            Auth_MySQL_User dbuser 
            Auth_MySQL_Password dbpw
            Auth_MySQL_DB dbname
            Auth_MySQL_Password_Table Users
            Auth_MySQL_Username_Field username
            Auth_MySQL_Password_Field passwd
            Auth_MySQL_Encryption_Types Plaintext
            Auth_MySQL_Authoritative Off
    </Location>

Microsoft Sho Word Histogram and the Python Standard Library

2011-01-27T19:00:04Z

Through a blog post from John D. Cook on planet python I beacame aware of the Microsoft Sho Project for data analysis and scientific computing. I haven't installed it yet, but it looks promising and I always love to see progress and usage of IronPython on Windows.

Here's the Computing a Word Histogramm Example:

>>> fp = System.IO.File.ReadAllText("./declarationofindependence.txt")
>>> table = System.Collections.Hashtable()
>>> for word in fp.split():
    if table.ContainsKey(word):
        table[word] +=1
    else:
        table[word] = 1
>>> pairs = zip(list(table.Keys), list(table.Values))
>>> pairs.sort(lambda a,b: a[1]<b[1])
>>> bar([elt[0] for elt in pairs[0:10]], [elt[1] for elt in pairs[0:10]])

I've used a hashtable for counting in the past too, but the Counter Datastructure from the python standard library (added in 2.7) is much better suited for this kind of task:

>>> from collections import Counter
>>> table = Counter()
>>> table(fp.split())
>>> pairs = table.most_common(10)

It's shorter and more readable.

For sorting a list of lists based on a specific element I prefer using operator.itemgetter instead of a lambda expression.

>>> from operator import itemgetter
>>> lst = [('orange', 5), ('banana', 7), ('apple', 2)]
>>> lst.sort(key=itemgetter(1))
>>> lst
[('apple', 2), ('orange', 5), ('banana', 7)]

The bottom line is that python has a great standard library and it is worth knowing it well.

Atom feed for Google Reader liked items

2011-01-06T06:29:29Z

While you can make your Google Reader starred and shared items public available, I found no way to do the same for my liked items. There isn't even a link from inside Google Reader to view your liked items.

But you can view your liked items if you open the following url http://www.google.com/reader/view/user/-/state/com.google/like. And there is even an Atom feed available http://www.google.com/reader/atom/user/-/state/com.google/like. The problem is that you have to be logged in to view the feed. So here is a small script to do a Google ClientLogin and dump the atom feed.

import httplib2
from urllib import urlencode
client = httplib2.Http('cache')

def request_auth_token(email, passwd):
    """request auth token from google

    see: http://code.google.com/apis/accounts/
         http://code.google.com/apis/accounts/docs/AuthForInstalledApps.html
    """
    url = 'https://www.google.com/accounts/ClientLogin'
    headers = {'Content-type': 'application/x-www-form-urlencoded'}
    body = urlencode({'Email': email, 'Passwd': passwd, 'service': 'reader'})
    response, content = client.request(url, 'POST', headers=headers,  body=body)
    assert response['status'] == '200'
    #if status == 403 check for Error=CaptchaRequired
    resp_data = dict(x.split('=') for x in content.split('\n') if x)
    return resp_data["Auth"]

def likes_feed(auth):
    url = "http://www.google.com/reader/atom/user/-/state/com.google/like"
    headers = {'Authorization': 'GoogleLogin auth=%s' % auth}
    response, content = client.request(url, 'GET', headers=headers)
    return content


if __name__ == '__main__':
    email = "XXX@gmail.com"
    passwd = "XXX"
    auth  = request_auth_token(email, passwd)
    print likes_feed(auth)

The same method should work to get the content of private folders in Google Reader.