Developing The New Herd

As you may, or may not know, for the last week, i’ve being trying to develop a new version for The Herd – the Ubiquity command aggregator.

I took this decision because the actual version does not work at all – mainly because we’re having too much problems with CouchDB (not that it’s a bad database, but our installation is behaving quite strangely).

The Herd today, is developed on top of Django – and after talking to Atul about this, i decided to drop Django entirely and use web.py.

My main reasons are:

1. Django is a full stack framework that makes you use their templates+their apis+lot’s of things – and for Herd, we wanted to do something like a REST-server with a html+css+js frontend
2. Django doesn’t have a good integration with schema-less databases (we *need* it for herd)

It uses web.py, MongoDB (it’s a great schema-less database) and simplejson for the server and plain html+css+jquery for the front-end (that i really need to improve).

It’s been 9 days now, and the new version is quite usable – it already fetches/parsers the feeds.
You can login (using openid) and comment on feeds (voting is almost complete too).

I hope i can “finish” it by the end of the week so i can really focus on the ui. (i will need some help on that).

Some screenshots:

Using Oracle Coherence with Grails Part I

Where i work, each with we’re assigned to a different tasks. My tasks this week are: test Beehive Webservices (i’ll post about it later) and test Oracle Coherence and Terracota.

I decided to start with Coherence because it really called my attention — it’s a in-memory jcache compliant clustered cache.

I started reading some docs about it and discovered that it can act as an 2nd-level-cache to Hibernate, so why not give a try with grails?

Everything is quite easy:

First, you have to download coherence zip to your machine and extract it. You will have a dir structure like this:

coherence
|– bin
|– doc
|– examples
`– lib

To start an Coherence console, just type bin/coherence.sh (with your JAVA_HOME set). It’ll show a lot of output, but the main thing is:

Services
(
TcpRing{TcpSocketAccepter{State=STATE_OPEN, ServerSocket=192.168.0.2:8088}, Connections=[]}
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.3, OldestMemberId=1}
)

Here it shows the cluster nodes — Coherence will always search for nodes on your machine through network broadcasting. So, if you start another console (on another machine, or on the same, you decide) you’ll see:

Services
(
TcpRing{TcpSocketAccepter{State=STATE_OPEN, ServerSocket=192.168.0.2:8089}, Connections=[]}
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.3, OldestMemberId=1}
)

Map (?):
2008-08-25 21:18:10.026 Oracle Coherence GE 3.3.1/389 (thread=TcpRingListener, member=2): TcpRing: connecting to member 1 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/192.168.0.2,port=50602,localport=8089]}


Here Coherence shows that it recognized the other node you started. (BTW on the other console you’ll see a message like this too)

If you type who on any console, coherence will tell you how many machines are on the cluster and who are them:

MasterMemberSet
(
ThisMember=Member(Id=2, Timestamp=2008-08-25 21:18:08.241, Address=192.168.0.2:8089, MachineId=26626, Location=process:30900@rome)
OldestMember=Member(Id=1, Timestamp=2008-08-25 21:16:35.527, Address=192.168.0.2:8088, MachineId=26626, Location=process:30378@rome)
ActualMemberSet=MemberSet(Size=2, BitSetCount=2
Member(Id=1, Timestamp=2008-08-25 21:16:35.527, Address=192.168.0.2:8088, MachineId=26626, Location=process:30378@rome)
Member(Id=2, Timestamp=2008-08-25 21:18:08.241, Address=192.168.0.2:8089, MachineId=26626, Location=process:30900@rome)
)

Now let’s start with the funny part (playing w/ Grails!)

First, you have to put coherence jars on your app — coherence.jar, tangosol.jar and coherence-hibernate.jar — this last one being the one with the 2nd level cache classes.

Then, put on your Datasource.groovy:


hibernate {
cache.use_second_level_cache=true
cache.use_query_cache=true
cache.provider_class='com.tangosol.coherence.hibernate.CoherenceCacheProvider'
show_sql = true
}


The bold part is the main thing — is the hibernate’s 2nd level cache implementation. I recommend you to put the show_sql to true, just to see when a query hits the database.

Then create a file called hibernate-cache-config.xml (ok, the actual name doesn’t really matter…) and put the following content on it: http://pastebin.com/f7c883504
This file contains the configuration on how the cache should be created, replicated and etc.

Now, put

static mapping = {
cache true
}


on the domain classes you want to cache. (this is important — and i didn’t know that, but thanks to Les Hazlewood’s explanation on this thread i know why i need it)

Now, let’s test it!

Fire-up a grails console instance with the following command:

grails -Dtangosol.coherence.cacheconfig=./hibernate-cache-config.xml -Dtangosol.coherence.hibernate.cacheconfig=./hibernate-cache-config.xml console


With this command you set two system variables (coherence uses them) that point to the file we created later.

After typing enter (duh!) you will see a lot of messages about coherence (just like the ones you saw when starting the console) (note: i recommend you not to have any other instance of coherence running when doing these tests). When the coherence finally started, save some instances of your domain class(ess) — i.e.:

import com.tangosol.net.CacheFactory

(1..25).each { new Author(firstName:"${it}", lastName:"${it}", birthday:new Date()-1).save() }

Then run this code on the same console (assuming your domain class is called Author):


import com.tangosol.net.CacheFactory

cache = CacheFactory.getCache(Author.class.name)

Author.get(1)
Author.list()

println "Cache size: ${cache.size()}"
println " ---- Cache Content ---- "
cache.each {k,v ->
println "Key: ${k} -- Value:${v}"
}

return null


On my machine, i see this output:


Hibernate: select author0_.id as id1_0_, author0_.version as version1_0_, author0_.birthday as birthday1_0_, author0_.first_name as first4_1_0_, author0_.last_name as last5_1_0_ from author author0_ where author0_.id=?
Hibernate: select this_.id as id1_0_, this_.version as version1_0_, this_.birthday as birthday1_0_, this_.first_name as first4_1_0_, this_.last_name as last5_1_0_ from author this_
Cache size: 25
---- Cache Content ----
Key: Author#1 -- Value:Item{version=0,freshTimestamp=1219712346919
Key: Author#2 -- Value:Item{version=0,freshTimestamp=1219712346928
Key: Author#3 -- Value:Item{version=0,freshTimestamp=1219712346929
Key: Author#4 -- Value:Item{version=0,freshTimestamp=1219712346930
Key: Author#5 -- Value:Item{version=0,freshTimestamp=1219712346931
Key: Author#6 -- Value:Item{version=0,freshTimestamp=1219712346933
Key: Author#7 -- Value:Item{version=0,freshTimestamp=1219712346934
Key: Author#8 -- Value:Item{version=0,freshTimestamp=1219712346935
Key: Author#9 -- Value:Item{version=0,freshTimestamp=1219712346936
Key: Author#10 -- Value:Item{version=0,freshTimestamp=1219712346938
Key: Author#11 -- Value:Item{version=0,freshTimestamp=1219712346939
Key: Author#12 -- Value:Item{version=0,freshTimestamp=1219712346944
Key: Author#13 -- Value:Item{version=0,freshTimestamp=1219712346945
Key: Author#14 -- Value:Item{version=0,freshTimestamp=1219712346947
Key: Author#15 -- Value:Item{version=0,freshTimestamp=1219712346948
Key: Author#16 -- Value:Item{version=0,freshTimestamp=1219712346949
Key: Author#17 -- Value:Item{version=0,freshTimestamp=1219712346951
Key: Author#18 -- Value:Item{version=0,freshTimestamp=1219712346952
Key: Author#19 -- Value:Item{version=0,freshTimestamp=1219712346955
Key: Author#20 -- Value:Item{version=0,freshTimestamp=1219712346956
Key: Author#21 -- Value:Item{version=0,freshTimestamp=1219712346957
Key: Author#22 -- Value:Item{version=0,freshTimestamp=1219712346960
Key: Author#23 -- Value:Item{version=0,freshTimestamp=1219712346961
Key: Author#24 -- Value:Item{version=0,freshTimestamp=1219712346962
Key: Author#25 -- Value:Item{version=0,freshTimestamp=1219712346963


What this means is that we have all 25 objects on cache! Uhuu!

On the next post….
– Add this to a real web-app
– Create a web interface to show what objects we have on cache and clean it

See you next time.

Auto Increment On A Oracle DB

This week i have a task: explore JHeadstart — Easy — specially if you follow their (uncommented) tutorial. But, when you start to build your own app, things start to get a little confusing (and, difficult) — specially because, by default, the framework creates screens with ids (the user can input whichever id they want) and this is not the right thing on a real-world-app. Ok, but if the user is not going to input the id, we have to create an auto-increment on the column.

Easy. Just put the auto_increment on the column…. No, not.
On Oracle DBs, different of MySQL, you cannot set an column to auto_increment, and it’s not easy to discover how to do it (if you don’t have an oracle dba in the house).

So, god google showed me a link telling me how to do it. If you don’t want to click on the link, i’ll show it for you here.

First, you need to create an sequence — the sequence is the ‘auto-increment-thing’, the one that’ll give to you the numbers.


create sequence funcs_id_seq
start with 1
increment by 1
nomaxvalue;


Then, you will have to create a trigger, that’ll be triggered when you insert an new row into the table:


create trigger func_id_trigger
before insert on functionaries
for each row
begin
select funcs_id_seq.nextval into :new.id from dual;
end;
/


Then, just to an normal insert into your table (w/o the id, of course) and the id will be setted!

Grails Auto Complete

Yesterday i finished my script for command completion on grails. It’s quite complete right now (not perfect, but almost complete).
I thought it would be more difficult to do it, but thanks to tutorials (here and here) and the other script for completion, it was quite easy.

The whole script (along with the installation and how-to-use-it can be found here: http://www.grails.org/Grails+Bash+Completion+Script

Ajude a sustentar a Wikipédia e outros projetos, sem colocar a mão no bolso, e concorra a um Eee PC!

…e também a pen drives, card drives, camisetas geeks, livros e mais! O BR-Linux e o Efetividade lançaram uma campanha para ajudar a Wikimedia Foundation e outros mantenedores de projetos que usamos no dia-a-dia on-line. Se você puder doar diretamente, ou contribuir de outra forma, são sempre melhores opções. Mas se não puder, veja as regras da promoção e participe – quanto mais divulgação, maior será a doação do BR-Linux e do Efetividade, e você ainda concorre a diversos brindes!

Adding Autocomplete to a Python Console

I really love python. It’s a great language (fyi it’s one of the languages that google uses — they are java, python, c and c++) and has a powerful console.
But, it misses an great point: autocomplete. (A think that ruby’s console has)

So, searching on the internet i found this site that show how to do it:

First, create a new environment variable called PYTHONSTARTUP pointing to a file “$HOME/.pystartup” — you can do that using the command export PYTHONSTARTUP=”$HOME/.pystartup. (You can add the command to your .bashrc, so you don’t need to type everytime you startup a console)

Then you create the file .pystartup and add the same content to it:

# Add auto-completion and a stored history file of commands to your Python
# interactive interpreter. Requires Python 2.0+, readline. Autocomplete is
# bound to the Esc key by default (you can change it - see readline docs).
#
# Store the file in ~/.pystartup, and set an environment variable to point
# to it, e.g. "export PYTHONSTARTUP=/max/home/itamar/.pystartup" in bash.
#
# Note that PYTHONSTARTUP does *not* expand "~", so you have to put in the
# full path to your home directory.

import atexit
import os
import readline
import rlcompleter

historyPath = os.path.expanduser("~/.pyhistory")
historyTmp = os.path.expanduser("~/.pyhisttmp.py")

endMarkerStr= "# # # histDUMP # # #"

saveMacro= "import readline; readline.write_history_file('"+historyTmp+"'); \
    print '####>>>>>>>>>>'; print ''.join(filter(lambda lineP: \
    not lineP.strip().endswith('"+endMarkerStr+"'),  \
    open('"+historyTmp+"').readlines())[:])+'####<<<<<<<<<<'"+endMarkerStr

readline.parse_and_bind('tab: complete')
readline.parse_and_bind('\C-w: "'+saveMacro+'"')

def save_history(historyPath=historyPath, endMarkerStr=endMarkerStr):
    import readline
    readline.write_history_file(historyPath)
    # Now filter out those line containing the saveMacro
    lines= filter(lambda lineP, endMarkerStr=endMarkerStr:
                      not lineP.strip().endswith(endMarkerStr), open(historyPath).readlines())
    open(historyPath, 'w+').write(''.join(lines))

if os.path.exists(historyPath):
    readline.read_history_file(maxhistoryPath)

atexit.register(save_history)

del os, atexit, readline, rlcompleter, save_history, historyPath
del historyTmp, endMarkerStr, saveMacro

Now you can start your python console typing python (Duh!) and try to do some autocomplete like:
a = “Fernando”
a.
a.is
a.islo

Enjoy!

Why i like Groovy

This is why i like groovy. I follow an brazilian java list, and last week, a guy posted some logic exercices in java. One of them was:

Given an array of numbers, create a function that searches all instances of a given number an print the number of occurences of this number.
Another guy posted the answer in java:

public class Modulo{
public static  int EncontraNumero(double[]  vetor, double  x, int count){
for (int i = 0; i < vetor.length; i++) {
if (vetor[i]== x)
count++;//calcula quantas vezes apareceu no vetor

}
return count;
}
public static void main(String[] args) {

double[] vet={2.3,5.7,9.65,3.33,4.10,5.23, 12.5,  12.5};
System.out.println(EncontraNumero(vet, 12.5, 0));
}
}

I created the same thing with groovy:

numbers = [2.3,5.7,9.65,3.33,4.10,5.23, 12.5, 12.5]
println numbers.findAll {it == 12.5 }.size

15 lines in 2. Wonderful

How to consume WebServices with Groovy

Primeiro instale o sdk groovy na sua máquina, usando o tar (ou zip) fornecido em http://groovy.codehaus.org/Download e instale como descrito em http://groovy.codehaus.org/Tutorial+1+-+Getting+started.

Após estar tudo ok, você conseguirá abrir um console de linha de comando digitando groovysh ou um console gráfico digitando groovyConsole.

Para realmente comecar a consumir você deverá baixar um jar da página http://groovy.codehaus.org/GroovyWS [TODO: explicar para que servem cada jar] e coloca-lo em algum lugar do seu classpath. (que pode ser em $HOME/.groovy/lib — , em $GROOVY_HOME/lib)

Voce pode já comecar a consumir webservices em wsdl, mas o jeito mais facil de comecar a fazer testes é criar um servidor wsdl local — para faze-lo, comece criando uma
classe de servico simples como essa:

class SystemService {
String ifconfig(lan){
def interface_lan = (lan ? lan : “-a”)
“/sbin/ifconfig ${interface_lan}”.execute().text
}

String top(){
“top -bn1”.execute().text
}
}

Essa é uma classe simples com dois metodos, que retornam os comandos top e ifconfig na máquina local.
Para abrir seu servidor wsdl, simplesmente rode esse script no mesmo diretorio que voce criou a class SystemService:

import groovyx.net.ws.WSServer

def server = new WSServer()
server.setNode(“SystemService”, “http://localhost:8888/SystemService”)

Pronto! Seu servidor já está rodando e voce pode acessar os seu wsdl pela url http://localhost:8888/SystemService?wsdl

Para realmente ver a “mágica” groovy funcionar, abra um terminal digitando groovysh e digite os seguintes comandos:

import groovyx.net.ws.WSClient
proxy = new WSClient(“http://localhost:8888/SystemService?wsdl”, this.class.classLoader)
proxy.top()
proxy.ifconfig()
proxy.ifconfig(“eth0”)

Essas linhas fazem exatamente:

1. Importa a classe necessaria para utilizar o webservice
2. cria as classes proxy on-the-fly utilizando o wsdl
3. Invoca o metodo top da classe SystemService definida acima na máquina do webservice server
4. Faz a mesma coisa com o metodo ifconfig (tanto sem parametros, quanto passando uma interface

A mesma coisa pode ser feita com outros webservices como http://www.webservicex.net/CurrencyConvertor.asmx?WSDL que é um conversor de currency bem fiel.

(Nota:  para conseguir os códigos das moedas desse webservice, você terá que executar:

symbols = Thread.currentThread().getContextClassLoader().loadClass(“net.webservicex.Currency”)
c.values().each { println it }

Já que a classe Currency é uma enumeracao, logo não pode ser instanciada, e nem utilizada diretamente, já que todas as classes necessárias para o webservice ser consumido foram criadas em memória.)

(Nota 2: Caso você queira compilar seu webservice server, para que ela seja “portável”, utilize o comando: groovyc SeuScriptWebService. Para rodar a classe resultante, utilize java -classpath /path/para/o/arquivo/groovy-all-1.5.5.jar:/path/para/o/arquivo/groovyws-standalone-0.3.1.jar:. SeuScriptWebServiceCompilado )

the Holy Grails!

What’s it?

Well, Grails is a web-framework based on a JVM (Java Virtual Machine, or just VM that’s how Jonathan Schwartz, Sun’s CEO is calling it) scripting language called Groovy.

Groovy?

I think is better to start explaning what’s Groovy.

Groovy is an scripting language that adds powerful additions to Java language like virtual methods, meta-programming, closures and lot more!

Why Groovy?

Well, Groovy is an excelent tool for Java programmers. Why have i said JAVA PROGRAMMERS and not just PROGRAMMERS?
I know that you have already heard of Ruby (and Ruby on Rails), Python (and Turbo Gears/Django) or even Scala (and maybe Lift). They’re incredible powerful scripting languages, and their frameworks are as good as them.

But they’re not Java. Their syntax not even looks like Java.

Ok, they all have counterparts on (J)VM, but… it’s not the same as programming in our favorite language.

Well, now imagine all the power of those language in a Java syntax. That’s Groovy.

A class in Ruby looks like this:

class Person
attr_reader :name, :address

def initialize name,address
@name    = name
@address = address
end

def rent_a_new_house address
@address = address if address
end

def to_s
“Name:#{@name} — Address: #{@address}”
end

end

For an guy comming from Java, that’s hebraic.
Now, if i say this’s a Groovy class:

class Person implements Comparable{
String name
String address

String toString(){
“Name: ${this.name} — Address: ${this.address}”
}

int compareTo(other){
name?.compareTo(other?.name)
}
}

The same guy cannot say he cannot understand this.
It’s Java.
But it’s not.

It’s Groovy!

Groovy adds a thing to Java called code by convention, a thing that came a long time a go in a far language… (that, i have to be honest, i don’t remember the name)

If you look to that Person class, you will say:

Where are the privates?
Where are the getters and setters?
Where is the constructor?
And WTF is that ToString?

Imagine something like that:

Everytime you create a class on your IDE of choice, you have to declare each attribute private, and them, see which one of them needs a getter or a setter. Then, you create a toString, and equals.

That’s boring, and still, you have to do it.

With Groovy’s coding by covention, to declare an private attribute along with its getter and setter all you have to say is: String name. or int age.

That’s it.

I know that there’re other things on that class that are not Java.
I’ll explain those.

First, toString:

String toString(){
“Name: ${this.name} — Address: ${this.address}”
}

The first thing you will notice on that method is there’s no return statement.
(The other thing is the ${} thing.)

On Groovy, as on another languages, you don’t have to explicitly say return name — all you have to do is add the thing you and to return as the last line of your method, i.e.:

def getName(){
this.name //return name!
}

The other thing, the ${} thing is called GString expression and it’s just like an JSP EL or and JSF EL. The variable name inside the curly brackets will be resolved and the string will be a lot prettier than something like:

“Name: ” + name + ” Address : ” + address

And with the benefit of calling methods insed the #{}!

There’s a lot more with Groovy. Too much, i think so from now, that’s it.
For more info: http://groovy.codehaus.org/

Stay tuned!

Adding Auto Complete to A Django-PyDev Project on Eclipse

Today i was trying to use eclipse 3.3 w/ pydev to develop an app with django, but i really couldn’t make the code-completion (the main reason (along with the debugger) i was using eclipse!).

The main thing problem was that i had to add ALL folders to the pydev project settings. And, i didn’t have time (and patience) to add one folder by one to the config screen, sooo, i created this script to do that for me:

#!/usr/bin/env python

import dircache, os

def listdr(_dir):
try:
dir_list  = dircache.listdir(os.path.realpath(_dir))
dir_list = [_dir + “/” + d for d in dir_list]
only_dirs = filter(lambda d: os.path.isdir(d), dir_list)

file = os.path.realpath(_dir)
print “<path>%s</path>” % file

for dir in dir_list:
listdr(dir)

except:
return

if __name__ == “__main__”:
listdr(“/usr/lib/python2.5/site-packages/django/”)

This script will show all folders under /usr/lib/python2.5/site-packages/django/ with that tags <path>, ready to add to the .pydevproject configuration file (that’s under your project file).