Libcoffee.net on Github

Posted by Jerry Wed, Oct 21 '09

Libcoffee.net's source code is now released at http://github.com/zanglang/libcoffee under the Creative Commons Attribution-Share Alike license, with which the end result is what you are reading right now. ;) The codebase is by no means complete, and by extension of that I'm pretty darn sure it's neither bugless, stable, nor razor efficient yet. However, it's being continuously updated so if by any chance you spot an error feel free to send a shout. So, to summarize:

Libcoffee.net is the name of the blogging engine [1] hosting this site. It's written using the Django framework, and is intended to be run on a Google App Engine account. Some of its features:

  • Basic blog authoring and publishing with plain URLs e.g. 'blog/2009/12/31/slug'
  • Comments with Google, OpenID and Facebook identities
  • Content markup with reStructuredText, Markdown and Textile
  • XMPP and email notifications for new comments
  • Clean default template with Blueprint CSS framework, hAtom and hCard microformats
  • Code block syntax highlighting
  • Trackbacks and pingbacks
  • Automated datastore backup via email
  • And of course, other miscellaneous benefits brought by running on Google's distributed computing platform.

There's a fair bunch of library dependencies used, so refer to READEME.rst for a list, as well as installation instructions. (So, uh, why yet another Django blog engine [2]? Answer: because it's not cool if you don't write one while learning Django. Amirite? ^^) All in all, fun experience. Lessons learned from this mini project:

  • Resource management on a distributed computing environment is very textbook. Disk space and CPU is cheap. Cloud computing, a buzzword it may be, could very well be the future of software. Memory is a lot cheaper than disk read/writes too, so it pays to cache the hell out of everything.
  • I have mixed feelings about running Django on App Engine, via app-engine-patch. Django itself is great, but BigTable imposes so many restrictions that half of Django doesn't work the same anymore, not to mention there's a massive requent-response latency when the app has to cold start [3], and then load the 3mb Django zip file, then crawl along as compiled Python bytecode is not allowed. In the first week, I was seeing ridiculously high response times averaging at 25 seconds each, but that seems to be have improved by now.
  • I clearly have not reached the point where I require the benefits of BigTable yet, so I frequently suspect if it would have been much less trouble if I just coughed up the money for a Python web hosting account... :P

And stuff on the to-do list:

  • Rewrite the comments/feedback modules. They were originally ported from Django apps, but a lot of unused cruft is still left over. I still don't know if trackback/pingbacks work properly, for instance.
  • Content caching is still very inconsistent. As it is, memcache keys are pretty much generated adhoc without a proper consistent namespace policy, and purged en masse every time a post is created so it's pretty wasteful. Need to get ideas from Nick Johnson recent posts on implementing a static content generator.
  • Use more of those leftover App Engine quota. I guess that requires me to blog more. :)
  • I really need to look at how to further cut down response times too. Perhaps it'll be yet another fun project - replace bits and pieces with newer and more interesting bits and pieces.

Related Links:

[1]I've originally wanted to call it Nii, as it's a *typo* of Ni, which was an utterance of the Knights from Monty Python, which inspired Python, which I would be using to write it with - uh, it was all supposed to be all connected and funny, okay?
[2]In fact, there's probably already a whole rainbow spectrum of blogs named Yet another Django blog in a dozen alphabetical variants, too.
[3]A proper analogy would be: after a period of inactivity an App Engine app goes cold (deactivated?), until the next request is received, where Google picks a new CPU slice for us to run on again. A popular app will propably stay warm quite well.
# Posted in 9 years ago comments
 

Fetching Twitter by App Engine Cron

Posted by Jerry Wed, Oct 21 '09

Snippet for fetching a Twitter feed to monitor for updates on AppEngine/Django.

from django.conf import settings
import feedparser # (http://www.feedparser.org/)

# cron job function
def update_twitter(request):
    TWITTER_URL = 'http://twitter.com/statuses/user_timeline/%s.rss' % \
                            settings.TWITTER_USERNAME # my username
    data = feedparser.parse(TWITTER_URL)
    if data.has_key('bozo_exception'): return HttpResponse('OK')

    pattern = re.compile(r'^http://twitter.com/.*?/statuses/(\d+)$')
    tweets = dict()
    for d in data['entries']: tweets[pattern.search(d['id']).groups()[0]] = d
    # there's a 'Tweet' model for previously saved tweets
    tweeted = [t.guid for t in Tweet.all().filter('guid IN', tweets.keys())]
    not_tweeted = [t for t in tweets.keys() if t not in tweeted] # there we go.
    if not len(not_tweeted):
        return HttpResponse('Nothing to save')

    for t in not_tweeted:
        ... # save t
    return HttpResponse('OK')

And schedule it in cron.yaml:

- description: Download twitter
  url: /lifestream/tasks/update_twitter/
  schedule: every 10 minutes`

Have I ever said I love Python's libraries? I love Python's libraries.

Seems that Twitter's RSS feed doesn't always generate successfully either (not unimaginable I guess, considering their concurrent hits), so it's important to catch errors... although they're not always very helpful (I've got several very generic 'Application Error: 2' so far, what on earth does that mean?). I suspect I'm not doing it very efficiently either, with 3 for loops and a dictionary, but not too many choices as I've only want to hit the AppEngine Datastore once for the entire query.

P.S - just noticed Markdown + syntax highlighting is still fairly dodgy with detecting whitespaces / code block boundaries. On to the TODO list it goes.

Edit: Attached a more complete code snippet. May have been a bad idea to blog at 1am.

# Posted in 9 years ago comments
 

Test post

Posted by Jerry Sat, Oct 10 '09

Test post, 1 2 3. Ping, pong!

    # testing code syntax highlighting
    def test(self):
        print 'lorem ipsum'
  • This is a list
# Posted in 9 years ago comments
 

Thesis code completed

Posted by Jerry Sat, Oct 27 '07

... Or so I hope?! Squashed a ton of bugs, stabilized API being used haphazardly between the RrdTool and Simulation sinks, sort of prettified Weathermap a bit more. Now I'm just left with churning out the rest of the thesis and getting it onto the NICTA testbed...

aesir:~/Projects/meshmon$ svn commit
svn commit:
Sending        html/index.html
Adding         html/overlib.js
Sending        html/weathermap.html
Sending        icons/Safemesh1.png
Sending        src/aodv.py
Sending        src/config.py
Sending        src/gatherers/rrdsink.py
Sending        src/gatherers/simulatesink.py
Sending        src/meshmon.py
Sending        src/nodes.py
Sending        src/plugins/RrdTool.py
Sending        src/plugins/Simulation.py
Sending        src/rendering/rrd.py
Sending        src/rendering/weathermap.py
Sending        src/threads.py
Sending        src/topology.py
Sending        src/webserver.py
Adding         src/wifi.py
Sending        weathermap/Weathermap.class.php
Sending        weathermap/weathermap
Sending        weathermap.conf.template
Transmitting file data .....................
Committed revision 61.
# Posted in 11 years ago comments
 

Summer of Code 2007

Posted by Jerry Sun, Apr 15 '07

It’s finalized: My Google Summer of Code 2007 proposals for Tomboy weren’t accepted. :( After results were out apparently there was one free GNOME slot that got unallocated and Sanford Armstrong offered to help, but it still turned out to be far too late. Pretty disappointed that I won’t get the possibly-awesome T-shirt or valid excuses to replace my laptop, but oh well. Going to pluck out original ideas from the proposals and start cracking on Gtk#. Expect some more Tomboy love soon, folks! ;)

Anyway, a review on this year’s SoC: Enthusiasts should definitely check out proposals listed on their favourite organizations. I have a good feeling about GNOME, Firefox/Thunderbird and Gaim Pidgin. Am I looking forward to the next few months of the Linux feature-fest. :)

And finally, going to release my Tomboy proposals into the wild under the Creative Commons Attribution license and move TomboyPIM into blog-post form so comments will be enabled (and search engines can index it, and maybe I’ll get more Tomboy/Google hits :)). Should be getting on it as soon as I finish my thesis report, due tomorrow (Ack!). Meanwhile you can check them out here: Networked Tomboy, and Using Tomboy as a Personal Information Manager: Integration with Evolution.

# Posted in 11 years ago comments