What’s all the buzz about curation?

Over the last couple of years there has increasing interest in the concept of curation, the idea that as the amount of content on the web expands exponentially, it may not be the availability of the content itself, but how it is organised and prioritized for you that matters (check out Google Trends on the growing use of the term ”curation”).

Whole new business models are springing up involving web sites who don’t employ journalists but rather act as hosts and curators of good quality blog posts.

A remarkable example in the financial world is SeekingAlpha.com where most of the good articles you read there every week have not been paid for by the site, and are written by people only loosely affiliated with it.

Added to that  kind of traditional editorial curation via websites is curation occurring in spades on social media services like Twitter, where by following experts or enthusiastic amateurs in certain areas, you have a high quality pre-selected flow of news delivered to you by people who you trust (but might have never met).

Personal curation

However the concept of curation that is of interest to me personally at the moment is (oddly enough) ‘personal curation’: content organised for my own personal purposes and not for an external audience.

The concept of personal curation is kind of odd really, surely with Google around the idea that you would want to retain and organise information for personal purposes is perhaps redundant?

There’s no need as you perhaps did 30 years ago, to print or photocopy documents that interest you and stick them in a filing cabinet never to be seen again. 

Why not simply search for them again? Or even just build bookmark lists?

In such a search-driven world a large part of your personal worth in the workplace as a domain expert, for example, might consist of your ability to understand the language used in relation to a certain concept, and with some ok syntactical search skills to be able find that information using the mother of all search engines.

Finding information does not seem to the issue. The personalised newspaper is here today and we’ve previously written about apps like Calibre that enable you to integrate the full text of literally any RSS feed out there on the web into your e-reader on iPad, Kindle, Nook, or whatever it might be.

Timeshifting your personal reading

You can also help organise information by ‘timeshifting’ the flow of links coming at you to save them for later  using something like the web-based bookmarking service ReadItLater (RIL), which will track which articles you have (and have not) read, and synchronise across multiple devices like your Desktop browser or your smartphone.

RIL works via Bookmarklets on your browser which add any pages you’re viewing to your personal list at ReadItLaterList.com (what we’re reading at Drivelry at the moment here), so you don’t have to interrupt your current piece of work to read that interesting item that just surfaced on Twitter. 

But what you do with an article after you’ve read it?  How can you retain and organise that information?

Retention of information you’ve read

Bookmarks, while easy to use, can rapidly fall prey to idiosyncratic folder taxonomy.  I’ve personally got 92 different folder categories under my bookmarks, and over 1100 favourites saved going down at least 4 levels.

Ouch.

The flow of email coming at you can be partially tamed (for retention and relocation purposes at least) through use of a full text search system running on your desktop such as Microsoft’s freebie Windows Search add-on.

My Desktop PC currently has almost 20 years of files and emails indexed this way via Windows Search.

An added benefit is that Windows Search will recognise sub-types of files and (providing you’ve used an appropriately keyworded bookmark name) enable you to search through your bookmarks as well, along with Office documents and even external network drives.

Saving e-book documents for later reference

On something like the Kindle you can ‘clip’ individual articles that you read to save for later.

However the Kindle saves them into a single text file which can be searched only very clusmily on the Kindle itself using the Kindle’s built-in keyboard.  You can periodically export it as a text file using Calibre and save it to your desktop where it will be picked up by Windows Search but as a single very large file on your Desktop PC it is not very useful for search purposes.

You can also use Calibre to automatically (say once a week) extract the full text of all recent articles in your ReadItLater list and load to any E-Reader.

Despite preferring the E-ink display of the Kindle over the backlit screen of say an iPad, one of the drawbacks of the Kindle is that if you see a link on the Kindle which you would like to read later  you cannot easily add it to your favourite social bookmarking service, and neither can you bookmark a current article so it becomes part of your personal searchable archive (see below).

In the medium term it’s likely therefore that I will give up using the Kindle for anything but books unless it starts to support this sort of functionality. 

Build your own Google Custom Search Engine from your favourite social bookmarking service

To be able to search the full text of  articles that you previously read and bookmarked there is another way to do it aside from saving the full-text of all web pages you read to your Desktop.

Essentially you export your entire ReadItLater bookmark archive and saved Desktop bookmarks and load them into a free personal Google Custom Search Engine (although you may not want the full-text of a lot of your Desktop bookmarks as they will often point to pages like your online banking application).

A reasonably straightforward process for creating a Google Custom Search Engine from your RIL bookmarks is described here (a 20 minute process to complete assuming a good working knowledge of say Microsoft Office).

Even better, by registering for the RIL Digest service (which produces a single web page with a magazine style layout of all your recent bookmarks)  you can point your Google CSE at the Digest and it will automatically crawl and index all your future bookmarks.

You now have  a cloud-based ‘memory‘ which learns as you read articles on the web. Drivelry’s memory is here.

This will enable you to search the full text of articles you have previously read and bookmarked assuming the web page is not removed, and automatically pick up future articles.

It takes about 20 minutes to do and requires merely good familiarity with Office apps and no programming skill.

The process should also be fairly similar if you are using another social bookmarking service. 

1. Obtain a free API key from ReadItLater: there is an  API request form on ReadItLater and it will be sent to you by email. You only have to do this to get your complete archive.
2. Export your entire bookmark list archive from ReadItLater using the command https://readitlaterlist.com/v2/get?username=name&password=123&apikey=yourapikey.  With the exception of the first time you do it you will normally just export your bookmarks since a certain date using the additional parameter &since=1245638446 where the number is a Unix style date-time format. For example: 1290795659 is 2010-11-26 18:20:59Z.
3. Save the resulting file as a text file to your desktop (which will be one continuous string with no line breaks).
4. Insert line breaks by loading the exported file into Word and search and replace “item_id” with ^l (the special linebreak character in Word).
5. Load the resulting file into Excel as CSV using the comma values in the files as separators.
6. Fix up the url slash characters in Excel by deleting other columns and then converting \/ to / again with a search and replace
7. Remove quotes on the urls by search and replacing :” with nothing and then replacing ” with nothing.
8. Assuming you have a Google user id go and create your free Google CSE at http://www.google.com/cse/manage/create
9. Enable automatic adding of future links by registering for ReadItLater’s digest service (current $5 p.a.) which puts all your recent bookmarks on one page (for an example of a Digest see Drively’s digest here at http://readitlaterlist.com/d/Drivelry) and set your Digest on ReadItLater to Public (if you don’t do this the Google crawler can’t see it). Add this digest url to your CSE using the Sites link in the Google CSE Control panel (CSE settings screenshot to right – click to expand) and select “Dynamically extract links from this page and add them to my search engine” and “Include all pages this page links to”.
10. Paste the resulting HTML generated into a web page you control. That’s it! You can see an example of what a Google CSE looks like here.

You can of course also export your personal bookmarks from your browser and also add them to your CSE. And you can make your CSE private or public as you wish.

Steps 1-1 are basically about getting your old RIL archive. If steps 1-7 look a bit daunting you can even skip them by just adding your Read It Later Digest page to the Google CSE and you will have all your articles added from that point in time (just not your older RIL archive).

Google’s limit for the number of individual urls you can add to a free CSE is 5000. For what it’s worth I add about 900 or so new urls on ReadItLater per year (which sounds a lot in aggregate but sounds less at 75 new urls per month) most of which are interesting links which come from people I follow on Twitter. 

For some other ideas and techniques see this article on curating web content.

The next information wave

It’s easy to hate the most over-used phrases used to describe the web age – paradigm shift, knowledge revolution, Web 2.0 etc.  

But hey we’ve all faced the tyranny of an empty page – it’s tough out there. And faced with amazing changes to how we locate information represented by Google and the other search engines it’s easy to reach for the bottle of superlatives.  

Even the growth of email has meant a phenomenal change to the way we consume and create information in the last 15 years as documents and comments flow back and forth in minutes that used to take faxes, printers, and letters hours or days.  

Despite all these examples of card-carrying “gee whiz”  moments however it looks like we are just about to be swept up in another wave.  

We are just starting to consume knowledge in another way which may result in an equally dramatic lifestyle change when compared to the growth of search engines, or the early days of the web itself.  

The on-rushing wave is the e-book (or more properly the ‘E-Reader’), with an estimated 10 million being sold by the end of this year just in the US alone and dozens of different E-Readers in the works from different companies emerging in the next 12 months – with a combined marketing budget in the hundreds of millions that will make the recent Apple iPad launch look like a hiccup.  

On the surface of it is this just another example of hyperbolic silliness? Because isn’t the E-Reader just another way to consume books that we’re already reading in hardcopy anyway, and as many have pointed out there are a lot of advantages of traditional hardcopy over the electronic alternative?  

What is interesting about E-Readers is not the ‘E-book’ part at all. Instead E-Readers are going to very rapidly reshape how we consume information on the web. This is not about a ‘new device’ it’s about a new way to consume information, because as is somewhat obvious most of the information on the web is just not available in hardcopy form at all. And that really matters.  

Backlit screen reading is hard to do (sorry Apple I know you’re trying with the iPad). It’s hard on your eyes,  it’s not very portable (try reading a laptop with your feet up), and in this information-packed age unless you’re looking at a huge screen it is hard to skim read and quickly identify the key points in screen-based information.  

This has lead to an emphasis on other aspects of screen-based information that compensate for these drawbacks of the web on a computer screen by adding new functionality: dense hypertext links to make cross referencing easy, the ability to comment and interact with authors as represented by blogs, the ability to locate and reassemble information in many different ways as represented by search engines, and in most journalism courses a recognition that writing for the web requires a specific style (admonitions to not write more than two sentence paras etc).  

But the E-Reader is changing that (read “Customized newspapers on e-books: Calibre cross-platform e-book subscription management” to see how you can subscribe to websites out there on the web on your E-Reader, via their RSS feeds,  just like you can subscribe to a podcast on iTunes).  

What happens when everything from the web is as portable and easy to read as paper?  

It’s not JUST the E-Reader: the way this is going to turn our information-consuming lives upside down is driven by a groundswell of the growth of machine-readable info feeds on sites represented by Really Simple Syndication, and emerging platforms like Calibre which enable you ‘subscribe’ to any website on any E-Reader – for FREE.  

The E-book just happens to be the final crest of the wave which is about to sweep away already undermined sections of the publishing industry.  

Still not convinced?  

The only thing worse than a ‘paradigm’ shift is an attempt to predict the future so here goes with a few near-term predictions:  

1. the world of journalism is about to get vastly more competitive – journalists in a newspaper will be competing with bloggers with more specialist knowledge who previously had limited circulation because you couldn’t purchase your average blog conveniently on a newstand or read it on the way to work on the train. The blogger is providing the information for free.
2. intermediaries that seek out big stories will get more important in their own verticals (wherever those stories are) whether they are using algorithms like Google or whether they are using human curated approaches like ‘The Week’, or social media mechanisms like Stumbleupon or Digg. Try and find for example, amongst the 100m+ or so out there which are the most entertaining blogs? To some extent this question was irrelevant  if those blogs didn’t get read – but not any more.
3. new ways of consuming stories will arise – for instance ‘collaborative reading’. On my Kindle or Sony Reader or iPad or whatever why should I not be able to view annotations on a text by my colleagues, or classmates, or friends?
4. we’re not talking individual wikis – we’re talking wiki world – why buy a ‘How To’ manual for  your motorcycle when there is a completely portable version?
5. traditional publishers who are congratulating themselves on being web savvy like the New York Times or Guardian are going to come under even more pressure because E-Readers in many cases will have the effect of destroying ad revenue and their ability to really track readership. With the aid of software on an E-Reader like Calibre this is the equivalent of Tivo for the publishing world: in your Kindle DX via Calibre no javascript is active (so Google Analytics aint going to know you’ve read the story) all it knows is that someone downloaded their RSS feed (not which particular stories were read).  And fancy ads are not going to come across because people writing recipes to extract your content with Calibre simply don’t want them.  Even if you instituted counter-measures by embedding an ad image joined with a story image, it’s almost the equivalent of going back to the flat ad serving done in newspapers. 
6. even the acknowledged guru of the web commerce world, Amazon, may be shooting themselves in the foot. They want you to buy e-books from them – but why? If you can read any old PDF or niche website out there on the web from the comfort of the couch? Or a million free book titles? I bought an Amazon Kindle DX and have yet to buy a book on it (but thanks Amazon anyway for paying the wireless subscription charges).
7. if it isn’t already obvious the E-reader is going to reduce further the cost of information globally even more: expect to add value with your startup company by better locating information, or better curation, but expect an uphill road if you think it’s about unique content (Mr Murdoch take note).
8. a bit like podcasting has broken the barriers to entry for local radio talk shows your decision is not ‘which paper do I buy in my home town’, instead it’s ‘which blogs do I want to read from the entire internet?’ Publications are becoming ‘unbundled’ and publishing deadlines paradoxically may actually become less important in an age of instant satisfaction. Why should I read some crap which some journalist stitched together at the last moment to make the Sun weekend edition when I can read a combination of bloggers who only produce pearls once a month? 

Like the sound of this new world or hate it?  

Is it not going to happen at all?  This is a blog – talk back.

What’s happening with internet-connected smartphones?

Looking at the current growth of smartphones where sales grew 50% in the 1st quarter of 2009 (year on year), now accounting for 1 in 4 of all phones sold, and particularly at the way that Apple doubled their share of the smartphone market with the iPhone,   it is clear there is something going on with huge numbers of people becoming ‘connected’ to the web while mobile during their every waking hour.

But what does that really mean? Is this change really significant? What is this kind of ‘always-on’ / ‘always-on-me’ technology actually used for and what kind of impact is always-on going to have on the group that uses it?

Well once people get smartphones (and particularly iPhones) this is what they do with them in Comscore’s mid ’08 European study of smartphone usage:

• 80% of users browse web-based news sources
• 56% use web search
• 32% watch mobile TV or video
• 42% use a social networking site or blog
• 70% listen to music
• 70% use email

In the broader mobile market Comscore reckons that more than 22 million people accessed web content daily via mobile in January 2009.

Cwipes! What a scary proposition! The newspaper industry has hardly adjusted to the need to rewrite and present copy differently online, now they’ve got think about mobile as well?

What is this kind of ‘always-on’ smartphone connected world going to be like for me as an individual?

Fortunately even before smartphones came along there was a demographic that was always-on. These are the guys who were born from1980: “Generation Y”. That’s 60 million+ people in the US.

Generation Y loves phones more than any other device.

For example, how many text / SMS cell phone messages do you send a month?

30? …. 50? …. perhaps you’d consider yourself a heavy texter who sends 100 a month (roughly 3 a day)?

Gen Y teens send 2200 texts a month.

It’s an absolutely stunning number of itself but it starts to make more sense when you do a little math.

As a Gen Y’er explained to me it’s just keeping in touch with about 10 people during the day with 7 messages sent over the course of the day to each person. 

“I’d feel naked walking into a cafe without a phone,” she said,  ”the first thing I do when I sit down with my coffee is text.”

Just as the Gen X’ers had trouble explaining how they were just ‘chatting’ on the phone to their parents (who saw the phone as something you only used for a special ‘transactional’ purpose), Gen Y use text messages more like status updates which can be broadcasted to multiple people in their phone address book.

The text message also carries a relatively low level of commitment. If a social event is being planned it’s possible to give only qualified approval by text, and then, because of the multi-threaded conversations going on, preserve your options to converge with whomever of your friends make sense in just a few hours time (read short term social planning). 

Difficult conversations and conflict are avoided by text but equally you may end up maintaining more ‘low-level’ relationships (notice how on Facebook your Gen Y friends have 200 or so Friends versus the Facebook average of  120).

Gen Y’ers of my acquaintance claim that they have more problems with friends who just won’t give them ‘attention-share’. 

 ’Status updates’? ‘Social committment’? It is all beginning to sound a bit Facebooky isnt’ it? Well according to Facebook 1 in 4 active Facebookers are accessing Facebook via  mobile device anyway (or about 60 million people).

Gen Y are just at the more intense end of an always-on curve that most of us are moving along anyway (50% of internet users now use Facebook) which is why Gen Y are worth watching.

How do I know when I’m becoming an always-on digital native?

Here are some ways to tell you’re becoming an always-on digital native, and how Gen Y seems to deal with the issues raised:

1. Feel like your Twitter Followers are more in touch with you than your family? Well Gen Y have been that way for a few years (unless their family are included in their updates!). Get into your Gen Y relative’s status updates!
2. Concerned about feeling out of the loop unless you’re constantly in touch with your friends? Don’t worry, unless you’re exceeding social contact with more than 10 people a day you’re just acclimating to the environment shared by the rest of  digital natives. Despite the fact that you’ve only got 50 Friends on Facebook you need to be aware that the numbers of close friends people keep in touch with is usually totally independent of this (most Facebooker’s only regularly exchange info with 5% or less of their Friends).
3. Want instant answers to any question? Gen Y’ers have grown up with Google since it really began to make waves around 2000 (the oldest Gen Y’ers were 20). They know how to use Google’s advanced search operators and they expect to instantly be able to find the answer to any question ‘of fact’ (analysis may take a little longer).  If you don’t know for example what the [site:] operator does on Google find out. 
4. Worried about the fragmentation of your attention when you’re trying to do original work? Well at least you’re thinking about it. 36% of UK broadband users (aged 16-55) state they have both the TV and Internet on in the same room every day. On weekdays the time when TV and Internet multi-tasking is most likely to happen is around 8pm in the evening (TNS/YouTube Media & Audience Study December 2008). Be aware that not every Tweet, IM, txt or email requires any level of committed response, and your spelling or grammar doesn’t have to be correct when and if you do respond.  Put off your next bout of Facebook voyeurism till the evening or weekend when you’ve got the time. At work why not cruise through the constant stream of interruptions with an uninterrupted iPod-driven music beat, set Outlook to pick mail once an hour, and get to know the web-based productivity tools out there like Pipes or RSS readers for cobbling together streams of information in structured ways that don’t all end up in your email inbox where they have to be dealt with.
5. Tempted to send an email or IM to deal with every eventuality? Gen Y’ers for all their obsessions seem to know the limits of electronic communication (perhaps those equally endless hours of movie consumption have taught them about body language?). Remember all that non-verbal communication and get out there and actually meet people. If you’re at work, call a project meeting so you can check out the negative body language of the guy who is not saying anything over electronic channels. Use the digital native tools that are out there to help arrange ‘face time’. Online dating is ok too: 1 in 10 American internet users claim to have used dating sites even back in 2006.
6. Thinking about what other people are thinking all the time? Living in an always-on culture with an always-present peer group can make you feel insecure when you don’t receive instant positive feedback on your latest choice of fashion accessory, and let’s face it someone has to do the jobs out there that don’t come with a support group (for instance being an HR Manager in the middle of a recession). A level of concern about Gen Y ‘neediness’ for approval is easy to find on the web, and in some cases has also branched out into a belief that creativity/originality might be harder for Gen Ys. It’s a concern that arguably appears unfounded: there’s quite a lot of evidence that Gen Y’ers strive to separate themselves from the herd, and are quite self-aware when it comes to dependency on others. Do both these things yourself.
7. Worried about privacy in the always-on electronic village? This seems a harder one to deal with. In this case you have the edge on Gen Y because you can remember youthful indiscretions that did come back to bite you! Think about the implications of privacy online, and use Facebook’s privacy settings in particular. For example, you probably don’t want your Facebook profile showing up above your LinkedIn profile when a prospective employer does a search and you even may not want registered Facebook users to be able to search for you if they’re clients as you don’t want to risk offending them by refusing their ‘Friend’ requests.

Always-on smartphones. Internet 1995 all over again?

A last point worth noting is that the 3G always-on world of mobile devices is not just something that society isn’t quite ready for.  Neither are the telcos. 

AT&T’s experience with handling the surges of traffic caused by iPhones has been bumpy. iPhones are basically the bandwidth-hogging Hummer of the smartphone world which can make the experience of using 3G connections patchy as the network is just overwhelmed (try getting through the online traffic jam whilst sitting in a tech related conference session with a couple of hundred other delegates some time).

The telcos are scrambling to catch up but at the same time they are already dealing with a rapid increase in the use of bandwidth-hogging video and audio – so the backdrop for them is challenging even when they’re trying to pass this stuff down wires rather than than through the air.

A great case in point is the film “The Jane Austen Book Club”. Despite the title you really don’t have to know anything about Austen to appreciate this film.

Written and directed by Robin Swicord its dialog sparkles along at a pace that it makes typical action pics look slow. With more layers than wedding cake you could watch it half a dozen times and still not pick up on all the jokes and nuances.

Ok it is a romantic comedy but it’s also a labor of love because behind a lot of the excellent jokes and gags are some serious and subtle points about relationships.

Labelled a ‘chick flick’ any man would enjoy the byplay in this film - there are a raft of muscular characters here and not just Sylvia’s gay daughter Allegra.  It is also full of surprise twists and shocks - you’d defy anyone to come up with anything so true-to-life awful as one of the opening scenes with Daniel and Sylvia in the restaurant.

Great film – terrible box office takings

 But the thing that is really most shocking about “The Jane Austen Book Club is how such a good film did so relatively badly at the box office taking only $7 million in revenue.

“Four Weddings and a Funeral” (which is a veritable pastiche compared to ”Book Club”) took $245 million….  Even Swicord’s previous film “Little Women” took $50 million.

Narrow it down to Austen literary territory and the comparisons look equally bad:

• “Bride and Prejudice” took $24 million
• Keira Knightley’s 2005 “Pride and Prejudice” took $121 million
• Emma Thomson’s 1995 “Sense and Sensibility” took $134 million
• and the  recent biopic about Austen herself “Becoming Jane” took $37 million

Perhaps one could argue that Austen fans like their adaptations as close as possible to the original based on the above but it is still interesting to ask why ‘Book Club’ did so comparatively badly (unless you think all of the above are so much better than it is).

What went wrong with this movie?

Here are a couple of guesses which have more to do with marketing than anything else:

•  why-oh-why did they call it “Jane Austen Book Club”? It’s like holding up a large sign to say “don’t come to this unless you are a) a woman and b) you ‘read’ with a capital R. Well, that’s 3/4ths of the filmgoing population gone then… What about Truths Not Universally Acknowledged? Read My Lips? The Club? Anything would have been better. Was Swicord so shocked when a producer said yes to her pitch (“I have this idea about a film about a book club…”) that she felt she had to stick with it?
• nobody seems to have really liked Fowler’s book amongst my acquaintances although I appreciate that somebody liked it in order for it to end up on the New York Times’ bestseller list for 13 weeks.  Was the film too closely tied to the Fowler book but actually aimed at a different demographic?
• did it need a more bankable star – would it have been the same movie with someone else as the female lead Jocelyn? It seems thoroughly unfair to even suggest this as Maria Bello played the role so well but would a Cameron Diaz or Michelle Pfeiffer have had more drawing power?  Would a better known male lead in the place of Hugh Dancy have delivered better box office returns – and again Hugh Dancy played Grigg’s role to perfection so it is unreasonable to suggest this on anything more than marketing grounds.
• did it need to be rewritten to appeal to a slightly younger audience? As a 16 year old said the other day “it didn’t speak to me” – with it’s acutely drawn portraits of middle age as delineated in the Daniel sub-plot did it just leave a younger audience too much in the cold?

We’ll never know the answer of course but someone should look beyond the box office results above and give Robin Swicord more money to make another film, this time with a better marketing plan.

The only strangely Austen-ish happy ending outcome of all of this was that the film was nominated for the Gladd Media Award and lost to another greatly unappreciated film which Drivelry has also reviewed, Stardust. The co-stars of the two films, Hugh Dancy and Claire Danes, announced their engagement this year.