Tag Archives: Development

Dev8D 2010 Review

This week I have been at Dev8D, a JISC-funded BarCamp-like conference which included a whole heap of three minute lightning talks from other JISC. Three minute talks are a wonderful format, long enough to hint at detail and short enough to skip boredom or irrelevance.

The best bit was of course my lightning talk, not because mine was “any good™” but because by doing it Jim Hensmen at Coventry University came up to me afterwards and introduced himself. It turns out that he is running an almost identical project to the Collaborative Tools Project at the University of York. Over lunch we shared our ideas and experiences so far, like Oompah Loompahs complaining about getting chocolate out of our uniforms.

This one meeting, for me made the whole trip worth it. Jim and I have arranged to share our directions and thoughts and results with each other.

Session: Yahoo Pipes

Tony Hirst gave a beginners guide to using Yahoo Pipes to scrape and shape data. I already follow Tony’s blog, Ouseful, and so didn’t learn anything new (that’s not a criticism) but I did manage to create something that I have tried and failed to do before ( a slightly complex aggregator publishing site ).

What I find fascinating about Tony’s work is that he has an almost maniacal avoidance of “programming”, creating things that “you can do too” without the need for programming skills. Added to that, he also has a unique ability to spot opportunities lying around the web, conceive of them in terms of tools like Yahoo Pipes, then he has the grace (like a bad magician) to carefully show you how he did it.

It was interesting to spot how many developers he’d infected with his mode of thinking, I kept seeing people Yahoo Piping all over the conference.

Session: Arduino

The wonderful Garry Bellamy, who isn’t a million miles from the Fast Show scientist who’s colleague was called Dave, gave us a beginners’ guide to Arduino. Arduinos are mini computers to which you can attach lights, motors, sensors that migt detect light/humidity/temperature or mini speakers, in fact almost anything, and make your own devices and robots.

I have so wanted to get my hands dirty with these for ages now, probably not because I will ever make anything but it’s nice to know what it’s like and what you might do. In our session I began to try and make our little and lovely LED light to blink in morse. I thought it’d be nice if a single light might be connected to a twitter account or search and tweet in morse, not so that anyone but a super-spy might read it but so that it could become an ambient “activity indicator”. Tweets tend to appear in an instant, displaying them via morse gives them more of a “visual echo” in that they simply take longer to “appear”.

Session: Amazon Cloud

Like playing with LEDs, I found this session extremely illuminating too. We created a computer in Amazon’s cloud, installed a webserver and MySQL, moved it onto a disk (also in the cloud) and then accessed it. What I appreciated most about this session was that it worked. Well prepared scripts were downloadable to do the things we needed to do (great preparation). We stopped when anyone got lost (for anyone read me). By the end of the session I had a number of servers running all sorts of services ( Alfesco, LifeRay etc).

Session: IPhone Development

This session gave us a introduction to iPhone development. I’ve tinkered before. I actually learned a lot, who’d have though THAT was the difference between header files and, er, implementation files. About three quarters of the way through the session my code stopped compiling and I couldn’t salvage it.

The thing that I think I took from this session is that I think I’m just too stubborn to accept crap programming language syntax. If the answer to a question why is “because you do” then my awkward gene throws a wobbly. I don’t think I will ever learn Objective-C and the world just probably be thankful for that fact.

Research Repositories

Throughout the whole event there was a buzz surrounding LinkedData and libraries and repositories. It’s ironic in a way that I found out about ePrints (a research outputs repository) is used at our university, from someone at our university (hi Aaron in the White Rose Grid) at the Dev8D event. Hey, I’m new…. still.

Other competing products/services that popped up in talks and mentions were Mendeley (ben.dowling@mendeley.com) , which I’d previously dismissed as an alternative to EndNote, a personal research bookmark-keeper but is more akin to LastFM’s audio scrobbler for research (it even has an open API). Interestingly, it tracks reading of research papers (how long I spent) so may have something to offer in terms of measuring the all important IMPACT factor in the future.

Mendeley and Zotero are things I will have to swot up on later.

Other sessions included:

Edina from the University of Edinburgh. Unlock Places a Geo Information Service. It includes hierarchical information… footprints, polygons, historical names of places

Twapperkeeper.com, an archiver of hashtagged tweets, brilliant for conferences.

Linked Data demos from Hugh Glaser, University of Southampton, http://rbkexplorer.com/demos/ http://sameas.org

Ross McFarlane @rossmcf gave an honest “lessons learned” introduction to Processing, the visualisation tool.

In the afternoon I learned Scala from Tom Morris, but, and this isn’t a criticism, Scala to me feels like Jython done different rather that Java scripting done right, but that’s just me, I only have room for a few languages in my head at once, including “body” and “english”.

Graham Klyne gave an insightful presentation on agility, demonstrating how the “travelling light” approach had lead them to avoid working with a web framework altogether (it was all done with Javascript, RDF, SPARQL, Lucene, Json, smoke and mirrors). An interesting assumption (which framework?) artfully and completely avoided by using gadgets (I guess).

All in all, a fantastic conference with the right mix of hands-on, sit and listen,have fun, get involved, just chill activities.

The catering was excellent. The whole event was well organised, friendly and helpful.

If I had a criticism it might be that some of the sessions suffered because the wifi really couldn’t handle a roomful of geeks downloading a development environment at once.

I do have  one suggestion …. I don’t think I quite “get” the whole LinkedData thing. Don’t get me wrong, I get the basics having noodled around with RDF/semantic stuff for years (very badly) … but there wasn’t a “get your hands dirty with the basics” session for LinkedData… I would have definitely enjoyed one of those, a session where I could go home with less than a page of working code that I’d written that maybe did something fun and simple with dbpedia and ePrints perhaps.

Django Dojo Eclipse Subversion Coda MacOS X

As web development frameworks go, Django really is fab. I can now run Django in a number of ways… all of which have their own advantages.

  • Development mode: The cool thing about running in development mode is that whenever you change any code, it automatically re-validates your data model and reloads it. Great for er, development. Any “print” statement is shown in the  Terminal. Code and reload…
  • Running under Lighttpd. Lighttpd is what my hosting company uses for web serving python things. What’s cool about this is that I can run a version in pretty much the same environment as on the hosting. I could use the Apache module mod_python on other hosts.
  • Running in Eclipse: Now this is clever… because I can set breakpoints, watch variables and generally step through code seeing what daft stuff I’ve done.

Eclipse is a big part of this, because the PyDev environment means, that not only do I have html code completion, but I get warned when I write bad python… oh and did I mention the debugging… it’s fab. Using print statements is fine when you have an easy bug, but I tend to write difficult buggers.
And then… if you get organised you can use  Coda to edit CSS style sheets live. Like the Firebug and EditCSS Firefox plugin… all fab…
I’m massively impressed with Django’s out-of-the-box CMS abilities. Although I must say, I want it to go further, to be more extendable, because, Django out-of-the-box is almost good enough for most CMS needs… I’ve just added the Dojo WYSIWYG editor into the admin screens… I can create new functionality for listing screens but not (I don’t think) standard edit screens, I may be wrong though….
And another thing… after dropping in the Eclipse Subversion plugin, not only can I do local development in a number of ways.. when I’m ready to upload to main server I simply do a “commit” and the main server is updated… just like that. I’ve been bitten in the arse a few times by Subversion’s finickitiness mind you.

So… what a collection of tools… if they weren’t so geeky they’d be usable. But for me the combination of python+django+eclipse+pydev+subversion+coda+dojo is moving towards something that is powerful and elegant… like a ballet dancer with a machine gun.

Nevermind the players, it’s the pitch that’s expensive

A friend, honestly, who shall remain nameless, told me recently that his web agency were onto the shortlist for a largish project… a shortlist of seven! Down from a staggering 14!

Now, of course it makes sense when buying in expensive services to show due diligence, cover your bases, research what’s on offer and see a number of companies, but 14 seems a bit steep to me, knowing the nameless company for which nameless person is pitching for.
In terms of time spent alone….

I imagine each of the agencies pitch documents would be read by at least one person.

14 x 30 minutes

The pitches themselves, I guess, might have 3 people present and take an hour each.

3 x 14 hours

The meeting afterwards to discuss which are the hot contenders would be at least an hour.

3 x 1 hour

Not taking into account the countless hours put in by the pitching companies time, in mythical man hours, that’s 52 hours, which is web agency terms is two weeks work (only kidding!) spent on the first round alone.
If you then imagine that the second round might be more detailed and involve more people, looking into figures and finances, asking for clarification etc. then it is reasonable to suggest that the time it takes for simply hiring the right agency for you could be well over a month.

A month’s work to hire an agency? I don’t think many people would put that much effort into buying other expensive items, like cars… not solid hard work. Even when buying a house, which can be a lot of work the idea of a month’s slog would put me off moving ever again. If you are commissioning an IT system in healthcare or planning a big corporate launch then of course, spending that amount of time makes relative sense.
What could be done to make the whole process of pitching more transparent, because, to be honest, if I was invited to pitch with 13 other people, I wouldn’t bother. A bit like a game of pitch poker, if the agencies involved knew who and how many they were up against, half would fold… which would be a huge cost saving for all concerned…

If the pitches themselves were open and transparent, perhaps a standard of sorts would emerge,, at least in terms of checkboxes that need ticking, to simplify the pitch into merely “What value add do you bring to project?”

It does make me think, as a company hiring a web agency for a large project, what would be a reasonable number of companies to see before you can make up your mind, knowing that you have made an informed decision?

p.s There is of course, especially in dot.com boom times, the deplorable practise of inviting huge numbers of agencies to pitch for a project in order to steal all the wisdom and good ideas presented. This is a scam I’ve heard about from lots of people in the industry and is typically used by the greedy and the clueless.

I once walked into a pitch with the same nameless person and he/she asked how many other companies were pitching. 12 they said, upon hearing he/she got up and walked out. I was kind of embarrassed and excited but it was an important lesson in knowing when you are wasting your time. Even if we had have been head and shoulders above our fellow pitchers the chances of getting the gig would have been statistically slim.

Anyway, good luck nameless pitcher! May their shortlists get shorter and still have you on it.

Python, SQL, Django shell

If you are anything like me, you poor fool, then although you love the database abstraction layer in django there are times when noodling around in python you’d like to quickly poke a bit of sql at your django created databases, wouldn’t you? You naughty, naughty thing… a bit like this perhaps…

    >>>from sql import prompt
>>>prompt()
sql:show tables;
((‘auth_group’,),
(‘auth_group_permissions’,),
(‘auth_message’,),
(‘auth_permission’,),
(‘auth_user’,)….. etc

sql:select * from music_chord;
((1L, ‘A’, 0L, 0L),
(2L, ‘D’, 0L, 0L),
(3L, ‘E’, 0L, 0L),
(4L, ‘G’, 0L, 0L),
(5L, ‘B’, 0L, 0L),
(6L, ‘Am’, 0L, 0L),
(7L, ‘C’, 0L, 0L)……etc

sql:describe music_chord;
((‘id’, ‘int(11)’, ‘NO’, ‘PRI’, None, ‘auto_increment’),
(‘name’, ‘varchar(200)’, ‘NO’, ‘UNI’, ”, ”),)

sql:exit
bye!
>>># Phew! Back into python after a quick peek into the bowels of the database

Well, now you can… drop this in your site-packages folder as sql.py

import os
from django.core.management import setup_environ
import settings

from django.db import connection, backend
from pprint import pprint

def prompt():
while 1:
sql = raw_input(“sql:”).strip()
if sql == ‘exit’:
print “bye!”
break
pprint (execute(sql))

def execute( sql):
cursor = connection.cursor()
cursor.execute(sql)
results = cursor.fetchall()
return results

Text Clouds: A New Form of Tag Cloud?

If the question is Text Clouds: A New Form of Tag Cloud? The answer is… they are exactly the same dummy! except of less value because lots of the words haven’t been carefully selected by humans, so you end with a lot of words like “the” and “and” and “where”. Now, of course you can remove a set of stop words but knowing when a stopword is a stopword becomes quite tricky. At first glance it seems easy, then you realise that there are words you want to exempt from the stopword list in this particualr case.

I’ve been doing some experimenting with “Text Clouds” lateley, creating one for a collection of competitiors’ web sites to help me see which keywords they are using most heavily and it has thrown up some interesting results. The good thing about them is that it gives you data that you can intuit… If it had have been a table most peoples’ brains would just fizz and make no sense of what is in front of them, but “Text Clouds” let you almost feel the differences.

Of course, Text Clouds-for-Phrases (two, three or four words) are where I’m heading. Because the word web most often hangs out with design and site, sometimes development. But then displaying web‘s relationship with both design and development becomes visually difficult. Should they be next to each other, overlayed… it’s harder than it sounds, trust me.

Maybe tag/text clouds are good at what they do because they are dumb and simple, forcing us to be more than smart, look at a collection of words, ignore the ones that don’t mattter and make some sense out of them, having had a smattering of help from a computer.
Imagine someone having watched the film of Ulysses a saying, “Yeah, but it wasn’t as good as the Text Cloud”.

If XML is so popular…

…and if if it an open standard… how come every time I come to work with it…

  • I can only find HUGE proprietary apps on (macos x)
  • I get an icky feeling in my stomach
  • Even the eclipse plugin kinda sucks

XML is fab until you actually want to use it… or is that just me?

What are mashups for anyway?

According to uninnovate in The Problem with Web Services, Google has suspended it’s Web Search API, which seems a bit shocking. Shocking in the sense of the fact that there’s a feeling in the air that service-oriented architectures, mashups and the like have recently become feasible. But the problem with this feeling is that it is based on trust. Trust that whole chunks of your application won’t disappear leaving you high and dry.

If it’s true that Google have axed their API, I’d love to know why. (It seems to still work for me though) Amazon have a similar API for which you pay micro-cents for, which so far, thanks to Kelvin and hi spython snippet, I’ve been completely happy experimenting with.

To be fair, Google’s API was always a bit rubbish, but to axe it only encourages good old screen-scraping, a way of getting the data that only takes an extra line of code anyway. Maybe that’s what they want? It would increase their usage statistics. I’d like to know that too.

But then, there’s lots I’d like to know about Google. Having accused me of clickfraud, then after apologising… my web site went from number two to LAST in Google on a search for “Tom Smith” (proud to be a number two!) Last?!?! Bitter? Me? Yes please!


But all of this leads to a question that Jonathan and I have been asking each other lately, and to be frank neither of us have any really compelling answers…

What are mashups for anyway?

The act of slapping a map on your web page is nice yes, but hardly worth mentioning. In fact the point that is worth mentioning is that often, the very map data itself has been collected and compiled and paid for by our taxes and then is only available to us tax payers if we pay huge licencing fees. Maps should have been free, online and friendly 10 years ago!

There have been a few nice examples of mashups, like the one where local paedophiles are shown on a map as living next-door-but-one to you. And yes, having some random pictures pulled from Flickr to look at whilst reading online will probably improve the experience in some way, for some people. But most mashups are crap, almost arbitrary and pornographic uses of technology in the “because we can” dept.

Back in 1998 I termed the phrase “para-siting”… to mean wiring together the ever growing number of services into a semi-cohesive whole, an environment in which aspects could be extended or de-coupled, say moving from geocities to lycos forums. I had a theory that quite fluid and complex things were possible. Now of course, my phrase didn’t catch on, partly because I’ve been a D-lister for longer than anyone can remember and partly because it was a crap phrase and partly because nobody knew what the hell I was talking about anyway.

And at the time people said… “You can’t ‘para-site’ an application because someone else controls your data and can shut it down, or be bought or simple lose it by accident”… which was a fair point…. because that is exactly what Google have done now. They have taken a large part of many peoples’ mashups and crippled them.

  • Most mashups aren’t mashups. They are online applications. Any mashup doing (or claiming to do) anything interesting isn’t a mashup.
  • Most mashups are the new viral advert. The mashups that take other data and display them differently (your delicious tags as an ants nest or bubbles or 3D space for example) are design adverts meant to impress you. You never return to these sites or use them but you may just refer them to someone else in the “something interesting” dept.
  • Most mashups are illusions. But they are selling the illusion we want to see. We want to believe that this online thing is all doable (totally) online, that it needn’t be a lot of work and that with a wing and a prayer, a dash of RSS and a little Ajax we can do something fab.

So, for a few seconds there I got quite excited about Tecqlo, Yahoo Pipes and Dapplets… and although offering the promise that anyone can make a mashup their own online application (the empowering aspects of which I am a huge fan of), at the end of the day, it’s probably not true.

There’s nothing I can do with pipes, teclo or dapper that I can’t do with python… and probably do it quicker too. And although I will have the hassle of running a server somewhere, I will have the ability to access my data, manipulate it, back it up, analyze it and make it look nice etc. Something most Blogger bloggers can’t do with their Blogger blog. If Google sets fire to the Blogger servers tomorrow… what then? How would a million poor spammers make their (huge) incomes?

So, whilst integration in itself can be seen as a “good thing” (I like the fact that the Flock browser works with Delicious)… integration isn’t everything (with the absense of substance)… which is as far as I can what mashups are for… to prove, in theory that two data sources can be joined togther. And having joined them together it normally becomes clear that, despite being cute, there isn’t really a compelling reason to join them together (but it seemed like a good idea at the time).

Pipes and Widgets

I don’t know if I’m late to the party on this one, but I hadn’t noticed Yahoo Pipes, which at first glance seems stunningly fantastic. I want to add my own sources though… For those of us that remember Rhubard and Custard, it’s a bit like AppWare … for those that don’t it’s a bit like Apple’s Automator for the web. It looks like I’ll be able to whip up a quick aggregator aggregator… amongst other things.
I made… A turn your blog into Spanish and back again pipe… and got…

 Dumb prescription of crepe of Tom Smith

…exactly… Take a look at this one too, from Nick Bradbury, which looks at the top 10 tunes (from iTunes) and shows related YouTube videos (also available as an RSS feed)…
Teqlo, the Mashup maker, logged me out, pissed me off, bombed my browser… and I love it! I tried to wire a Google Spreadsheet of York pubs to a Google map… and failed… but almost got there… A tool has to be very, very promising indeed to be that bad and yet keep me hanging on on
And I think the two tools above can be made to talk to each other. I like both approaches. Pipes seems to be about the back-end… information input and output… and Teqlo seems to be about the front-end…

Visual programming is creeping up on us slowly… and once you have a great user interface able to deal with pipes and widgets, which is all most online applications are… deep down at least… it opens up the web to millions of more information authors, able to make the web work they way they want… which is liberating and wonderful… and also very, very scary…

Open source rocks/sucks! Deal with it :-)

I think Jason misses a few points in Why Programmers Don’t Trust Other Programmers (via dzone)
Code is a Verb not a Noun: Reinventing the Wheel is a Good Thing… especially if you are adding tyres. Reinventing wheels is about learning about wheels, learning about motion, not about churning out another wheel. In fact when teaching a new programming language people tend to use something called a Fibonacci which I think is a flavour of ice-cream, not because it’s a useful thing to do, but because it’s a recognisable wheel.
Standards not Products: Open source products tend to suck. Often not less than the commercial counterparts. Hell, lots of software sucks. Where open source (for me) really flies is in the area of standards… such as XMLRPC, unix, HTTP, PHP(ugh!), Python(yay!), HTML, SQL. Yes there are some great open source products such as WordPress and Eclipse, but the best open source tools are low level… like some C code that resizes pictures. Now there’s a problem I would like, or be able to solve myself.

The thing is… as long as I can interface to something in a recognized and standard way. I’m then free to recreate my own wheel as badly as I like. Maybe even making a raft from inner tubes or something. I’ve done worse…

Ideas not Standards: Of course, when you use any standards you find out there’s no such thing as a standard standard and it tends to be the ideas or concepts behind them that are the things with the true value. Ideas such as object-orientation (not bad), templating (very practical), versioning (a good idea), group moderation (how cool) which, when wired together using standards, is guaranteed to succeed… how could it fail?
Expectations: If you are looking to integrate 2 complex systems, it probably won’t work. If you expect people to have created a true component-based modular system that you can bend to your needs with 2 lines of code, grow up. One of the best attempts made at this in my experience was Zope and yet very rarely could plug-ins (or Products as they were called) be easily integrated. The best one could do was…well, often pretty shoddy. ( Having said that, have I said how totally stunning Eclipse is? I’ve been using it for a fortnight and it is the best MacOS X Python/HTML editor ever! All the plug-ins work! Amazing!)
The Experience Trap: In order to assess a system, you get corrupted. A great example here is open source content management systems. Is it me or do they all suck? I have tried loads and I pretty much hate them all on grounds of usability. But the problem comes when, to trial a CMS you have to live with it, sometimes for years, you have to hack it, reinvent wheels, integrate standards based bits and bobs, worth with their plug-in architecture, by which time… you are now adept. And swapping being adept for not being adept, learning a new system needs some bloody big promises to be worth it. Computing power corrupts.
Abilities: Most of the open source software I come across is written by people with more ability than me, but they don’t have my sensitivities. When I have complained or suggested certain usability features the reply I have universally received is, “great idea, now go do it… here’s the cvs server”… In order to work well with an open source project, you normally have to understand it in its entirity.. which if I could do, I would probably disagree with it on architectural grounds and fork my own version…. perhaps.
I haven’t been a D-list blogger for 7 years for nothing. The open source community doesn’t listen to good ideas or suggestions, it listens to actions (or code).
The Boringness Factor: The “desire to be solving problems” that you talk about has a lot to answer for. For example, because someone has “solved” Social Networking sites, there are probably dozens of versions out there that I can download and have running in an hour or less. Now, unless I was bringing something extra to the table, I personally can’t get excited about solving that particular problem. So, I’d lower my expectations and run with an existing version, sure it may not be modular or as usable as I’d like but it would hopefully have some code for handling round-trip emails, and forgotten password tools and all those other things that every site needs but which, if possible, I’d rather not write again.

And, in certain circumstances, lowering your expectations has advantages… such as.. having plumped for an open source Social Networking site, I will, as long as I don’t hack it too badly, benefit from maintainance updates, something no real programmer likes doing 🙂

I’m both a total devotee and hater of open source software. I use a Mac and love Python.
This week, as I’ve said, I’ve been massively impressed by Eclipse (which after about 5 years is now fast enough for my MacBook Pro to be usable) and OpenWYSIWYG (which is a good start for web rich text editing… about time!!!!) And I’m old enough to still remember the days when, having posted a project-killing but simple replicable bug to Macromedia, I would hear that they were “right on it” and that it would be “fixed asap… in the next 6 months”… ouch! ouch! ouch!
Open source rocks/sucks!  Deal with it 🙂

How to get from “fishes” to “fish” in python

Thank goodness, calls to make words CamelCase and singularize plurals in Inflector.py. Thank you, thank you… searching for something to do this has been very, very difficult. I’ve also been struggling with sorting dictionaries which to me should be the way you do things… complex data structures need to be sortable… I lost an hour today trying various lambda do-das… which always confuse the hell out of me (but/and fit on one line).

Object-orientation… I’m working on a project at the moment, one I’ve tried to do a few times before and always failed. I’m not sure why. What’s interesting for me is that because Django is soooo M-V-C, in order to build each component I’m having to test it using Django’s shell. And what that means is that the model (M) works on the command line before I start slapping an interface, or the view (V) on it.

This make the code work before you start working with it approach goes against the it’s not till you see it until you know what you are trying to make approach … but I’m liking it.
It’s forcing me to understand object-orientation, which I’ve been a fan of for over 10 years, and start to know where code should go. In the past… the view (V) has always got muddied with the model(M). Add to that my personal preference to be able to run certain modules standalone so that I can test ’em to death without the overhead of a framework… and I really feel I’m getting onto a next level of sorts.

Now, if someone could explain the what the C is, I think I’ll be laughing.