The rise and rise of data

Even a cursory study of the history of the internet, especially after the advent of the World Wide Web, indicates the power of the medium. In terms of users, it has grown 588.4% between Dec 2000 and June 2013, from around 369 million to 2.4 billion, though only 34.3% of the world's population had access to the internet in this period [source]. The penetration is expected to grow to 45% by 2016, bringing the digital world to almost 3.4 billion users.

Businesses and governments have taken advantage of the internet, especially of the WWW, to create applications that many of us can't do without. Shopping for most of your needs, reservation of your airline, train, bus tickets, planning and managing your holiday tours, most banking activities, paying your bills, buying and renewing your insurance, filing your tax returns - all of these can be done over the internet. Streaming music and video keeps us entertained, social networking applications and blogs fulfil our need to share, Skype and WhatsApp help us to connect. Google Drive, Onedrive, Dropbox store our important documents and allow us to share restrictively. Almost every sphere of activity have applications dedicated to it, photography, ornithology, skill enhancement, stocks, funding of ideas to production, you name it, all allowing us to generate and use content.

All of these generate data, huge volumes of it. The infographic in this article gives an indication of how much user driven data is generated every minute. CISCO estimates total Global IP traffic by 2017 will be 120643 PB per month up from 43570 PB per month in 2012.(What is a Petabyte (PB)? also see here).

Not all data is user generated.

A example is India's 'Adhar' initiative, implemented by the Unique Identification Authority of India [UIDAI]. Briefly it aims to provide a 12 digit unique identification number to every Indian (that's over a billion people) starting August 2009. It's mandate is to provide these numbers to 600 million (60 crores) citizens by 2014. It enrolls a citizen by collecting his/her iris and thumb scans and demographic data. Each enrollment pack is 5 MB of data. Till now it has generated 1500 PB of data [source].

Every department of governments generate a humungous amount of data. Analysis of these data create more data, usually in the form of reports. Government rules and regulations mandate business to generate and publish to the public domain their Annual Reports. Organisations like Lexis Nexis collate business data from the public domain all over the world and create tools such as their Dossier Suite.

Look around you and everyone is creating data and very large volumes of it: the United Nations, stock exchanges and financial and credit rating organisations, researchers and scientists, even machines and processes where sensors report the working parameters at regular intervals.

The question then is: How do we persist and maintain large volumes of data? And how do we use these? We’ll check these out in subsequent posts.

Commodity hardware

One of the USP of NoSQL databases is their ability to run on 'commodity hardware/ machines/ servers/ clusters'. So what is commodity hardware?

My desktop PC is powered by an Intel i5 4th gen processor, has 16 GB (initially 8, upgraded later to 16) of RAM and 1 TB HDD. The hardware is reliable and affordable, even though it was built by an assembler. It is easy to upgrade when required - like I did with the RAM. I have no vendor buy in; my first preference for an upgrade would be the current vendor, but it is not necessary for me to buy from him. I am also not too concerned about the make of the components - after all the industry has matured to an extent where there is not much to distinguish between different makes.

The same would be the case with a branded desktop that you may have bought. You may have vendor buy in because of the warranty, but nobody really can stop you from adding memory or storage or a game card, or maybe even replacing the motherboard and the CPU.

This essentially is then a commodity machine:
Affordable, reliable and upgrade-able
No (or limited) vendor buy in

A commodity cluster is made up of commodity machines working in parallel to increase computing power. Serious supercomputers have been built this way; the nitty-gritty of building and maintaining clusters are known. In the NoSQL context, clusters increase performance and provide scalability. Add machines to the cluster if demand increases; maybe even remove a machine if the demand reduces and do these without disrupting services.

A start-up from Bangalore has built a data center using commodity hardware. A case study worth reading.

Creating complex business solutions

In spite of the fact that I haven't posted on this blog for 2-2.5 years, there has been an average of 100 page views a month. I assume from this that the content has been helpful to some. Inspiration enough to start again.

Technology has made giant strides in these few years making it possible to create very complex business solutions. Two areas get highlighted:
a. how do we maintain large volumes of structured and unstructured data, query and analyse it
b. how do we build and deploy applications in terms of performance, scalability and mobility.

It is a time when developers are under extreme pressure to learn and understand a lot of new technologies (in addition to the traditional ones) in a very short time - and deliver. .NET developers, IMHO, are at a greater disadvantage, because solutions are more in the Java/ Open Source space and require more effort on their part.

In subsequent posts we will discuss the topics below. This will address some of the areas related to 'a' above.
1. NoSQL databases: the need, different types, performance comparisons, applications
2. Search: the need, different approaches, performance comparisons, applications

IE's legacy browsers - will they ever go away?

“W3C publishes documents that define Web technologies. These documents follow a process designed to promote consensus, fairness, public accountability, and quality. At the end of this process, W3C publishes Recommendations, which are considered Web standards.” from Standards FAQ.

The recommendations are not binding, but as the W3C says: “W3C's role in making the Recommendation is to draw attention to the specification and to promote its widespread deployment. This enhances the functionality and interoperability of the Web.”

And would make us developer’s creating web applications more productive.

Microsoft, in spite of its large representation in the W3C Working Groups, has been a laggard at proactively adopting these recommendations in its browsers. It has thus gifted us a bunch of what it now calls its ‘legacy browsers’ - IE6, IE7 and IE8, which, whether we like it or not, has still to be supported if our web applications/sites are to reach a greater audience.

A case in point is the CSS Color Module Level 3, one of the modules of CSS3, which was endorsed as a W3C Recommendation on 7 June 2011. The first W3C Working Draft was published on 22 June 1999, it became a W3C Candidate Recommendation on 14 May 2003, back to a W3C Working Draft on 21 July 2008, and a W3C Proposed Recommendation on 28 October 2010.

Tantek Çelik [Microsoft’s representative till 2003] and Brad Pettit [Microsoft’s representative] have been associated with this recommendation right from the first working draft to its endorsement as a recommendation.

In this period, IE went from IE6.0 to IE9.0, Firefox to version 5.0 and Chrome to 12.0. The table below indicates how Firefox and Chrome were early adopters of CSS3, whereas Microsoft waited until IE9.

Table 1

How does Microsoft’s apathy affect us in development? Several ways:

1. IE is the most used web browser.

Figure 1: Web browser usage by country in September 2011 [Source: http://en.wikipedia.org/wiki/Usage_share_of_web_browsers]

2. StatCounter estimates the usage share of web browser for September 2011 as below.

Figure 2 [Source of data: StatCounter]

3. The usage share of various versions of IE for September 2011 is shown below (source: StatCounter)

Figure 3 [Source of data: StatCounter]

Figure 3 means that the usage share of Microsoft’s legacy browsers is 32.72%. This is a large population and cannot be ignored.

So why do we have a such a large population using IE’s legacy browsers? 2 reasons in my opinion.

One: Microsoft bundles IE with its OS. So IE is kind of available out of the box. The fact remains that the majority of Internet users are not techies or geeks like you or me. The browser is just another tool to get their work done. And if it is available out of the box, they’ll use it. They don't care if the IE version they are using is W3C compliant or otherwise, as long as it can get their work done.

IE also has a tight integration with the OS. So while you will be able to install Firefox 7 on Window XP or Window 7, IE9 will work only on Window 7+ (though its legacy browsers can run of this OS). The high usage of IE8 is because it was released for Windows XP, Windows Server 2003, Windows Vista, and is the default browser for Windows 7 and Windows 2008 R2.

Window XP and Window 7 more or less share the honours as per StatCounter’s data of September 2011. This delays adoption of IE9 as the primary browser.

Figure 4 [Source of data: StatCounter]

Two: Figures 1, 2 and 3 indicate internet usage. My experience is that the usage of IE’s legacy browser is even greater in intranets in corporate, business and government - this follows from the fact that the browser is bundled with the OS. These have a large user base and the cost and time of upgrading the OS or the browser is huge.

So, IE’s legacy browsers aren’t going away soon. And we will continue to fret for a while longer about the pain these give us. And because we create applications for the masses, we need to write code to support both these and the modern browsers. Something like this ‘best practice CSS’, even though it will never validate.

.50PercentOpaque
{
    opacity: 0.5;     //W3C CSS Color Module Level 3 compliant browsers
    filter: alpha(opacity=50); //IE legacy browsers
}

Why I did not buy a Samsung tab...or an iPad

Last Saturday, I walked into an electronic store to purchase a Tablet, a tab. My preference was for the Samsung 10.1; in the store I also came across the iPad 2. And in this section was also on display an array of laptops.

Why do I need a tab? My usual practice is to surf the net after dinner, possibly read an e-book, catch up with people on Facebook or LinkedIn. I do it on a laptop connected to the net through Wi-Fi, sitting at a table in the living room. And I doze off most of the time. It would be so much more comfortable if I could do all of these propped up in bed, and doze off without a care.

The Samsung 10.1 on display was a 3G+ Wi-Fi 16GB version priced a little over Rs 33,000. Its cover, a necessary accessory, was an extra Rs 3,500 or so. Which meant that the tab would cost me between Rs 36-37K. The 3G+ Wi-Fi 16GB version of the iPad2 with cover was Rs 37,000. The one with 32GB was around Rs 41,000.

At this point I began to wonder if, at this price, it made any sense to buy this device. At a price of Rs 40-45K one could purchase a much, much more powerful laptop with oodles more hard drive space, and do more on it than is currently possible on a tab. [Don’t believe me? Check out the Dell Vostro 3750 which sells for about Rs 41,000 all inclusive.] And of course, any worthwhile laptop can connect to the net through a Wi-Fi connection, or a wireless device from your favorite mobile service provider, or can be tethered from a 3G phone. So what would I miss? The touch screen for one, its light weight, and certainly the convenience of surfing or reading propped up in bed.

Didn’t I know all this before I visited the store? I guess I did, but the heart went like “Let’s get one, let’s get one”; and it was only after we actually fiddled around with the tab that the head asserted “Come on. This gizmo is not worth 37K. A 40K laptop is more value for money any day. Get a hold of yourself. You want to spend 37K so you can surf lying down, and push things around the screen with your finger? Shame on you!” The heart didn’t have an answer.

So I am back at my laptop. The heart is disappointed. And so is the head. This innovation did not match up to expectations; or perhaps it wasn’t created for me.

India: my pride, my shame

Today India completes 61 years as a Sovereign, Democratic and Republic state. We have been an independent nation for 64 years.

In this short span of time India has perfected the art of being corrupt. The CWG scam, the 2G scam, the Adarsh scam, $500 billion dollars of the country’s money stashed away in secret accounts abroad, bureaucrats discovered with Rs.300 crore stashed in suitcases at their home - these are the ones that are being talked about now. There are many from the past that we have forgotten. More importantly, we don’t remember anyone of any importance being ever punished.

With such a capacity to indulge in it, it is no surprise that corruption is so pervasive in our society, even soiling venerated institutions like the Armed Forces and the Supreme Court. We don’t need Transparency International’s studies to tell us we are rotten.

As a nation we need stalwarts in public life to look up to. The picture below defines stalwarts. We don’t have people of their vision, stature and influence anymore; however, we are desperate for them, gauging by our admiration for people like Dr. Abdul Kalam and Narayana Murthy. We need more and more people like them; we need their influence over us.


[Photo source: http://en.wikipedia.org/wiki/File:Gandhi_and_Bose.jpg. Don’t know who they are? See here.]

India has many good things going for it. Game-changing activities in the last 6, mostly in the last 3, decades in areas as varied as computerisation of railway reservations, telecom, low cost airlines, software and manufacturing, space research and technology, media, freeing the economy, strengthening the Election Commission, have taken our nation forward. These have been achieved by ordinary citizens doing extraordinary things without funfair, helped by some good thinking by those in government.

We need to do more: in infrastructure, providing quality education, improving agriculture, maintaining the unity without diluting the diversity and, our eternal bug bear, eradicating poverty. And there are many, with insight and zeal, already doing their bit in bringing about these changes. The scale is huge, given India’s diversity, size and population - and this all pervasive corruption makes the work tougher.

To be counted as an advanced nation, a nation truly respected, we must eradicate corruption from our midst.

Help...

http://www.saveourtigers.com

Where the mind is without fear ...

"Where the mind is without fear and the head is held high;
Where knowledge is free;
Where the world has not been broken up into fragments by narrow domestic walls;
Where words come out from the depth of truth;
Where tireless striving stretches its arms towards perfection;
Where the clear stream of reason has not lost its way into the dreary desert sand of dead habit;
Where the mind is led forward by thee into ever-widening thought and action…
Into that heaven of freedom, my Father, let my country awake."

- Gurudev Rabindranath Tagore, Noble Laureate, Literature, 1913

Tagore's poem is a prayer of universal appeal. It is also a vision and a mission.

It is a vision, a mission and a prayer not only for my country, but for all mankind, especially in these times of conflict all over the world; replace the 'my country' with 'me' and it is a vision, a mission and a prayer for each one of us.

India Leadership Forum 2010

Early in Feb, a colleague and I, sponsored by our company, attended Nasscom’s NILF 2010 at Mumbai. To say the least, it was a wonderful experience. Nasscom has uploaded most of the video recordings of the sessions here. And you can read their blogs, which provide a synopsis of the various sessions, here.

Many industry heavy weights were speakers, but, IMHO, the academicians stole the show at this meet with their very incisive insight into the future. Their presentations were lucid and masterful. I list below some of the sessions I attended. I urge you to go through these presentations.

Building a Learning Organisation by David Garvin, Harvard Business School
synopsis: http://indialeadershipforum.nasscom.in/blog/2010/02/building-a-learning-organisation-workshop-by-prof-garvin/

Expect (and welcome) the unexpected - a new approach to strategy by Lynda Gratton, London Business School
synopsis: http://indialeadershipforum.nasscom.in/blog/2010/02/expect-and-welcome-the-unexpected/
presentation: start with the video labeled Session VI B1 here.

Lynda Gratton also held a work shop titled Get Set Go…Building Organizations With a Vision.

Standing out in the crowd: what’s your differentiator? by Deepak Jain, Kellogg School of Management
synopsis: http://indialeadershipforum.nasscom.in/blog/2010/02/standing-out-in-the-crowd-what%E2%80%99s-your-differentiator/
presentation: start with the video labeled Be a Phoenix 1 here.

Is innovation a priority in challenging times? by MS Krishnan, University of Michigan
presentation: start with the video labeled New Age of Innovation 1 here.

The new rules of marketing and PR by David Meerman Scott, Freshspot LLC
presentation: start with the video labeled New Rules of Maketing and PR 3 here.

Some words that we heard across the sessions - innovation, eco-systems, demographics, culture, value systems, cloud - summed up the focus of this meet.

Looking back at 2008

2008 was a good year. Once I got settled in, the last year at my current job has been very satisfying.

I lead the effort on three AJAX projects using the ExtJs Javascript library. Learnt a lot about code optimisation and performance in AJAX projects, object oriented JavaScript, JSON, stuff like the CTE in SQL Server 2005. The client's comment "..one of the fastest web applications I have used.." validated the hard work we put into the projects. I have begun to post my experience with AJAX in the A for Ajax posts in this blog. There are 3 posts on this subject already, more will follow.

There was another interesting project - a BI application where the data was being presented through web parts - where we got to investigate more asynchronous activities - like asynchronous loading of the web parts and asynchronous calls to the database. The technical demands of the project kept us on our toes, but the project was ultimately scuttled by bad client behaviour.

The last year has also been about earning the respect of my colleagues at my current job. I have always been blessed with very good colleague work throughout my career: they work hard, they take responsibilities willingly, they have a hunger for knowledge, are open to new ideas and ways of doing things, they create the magic in the end product.

A very interesting aspect was the diverse clientele I had to deal with - across Australia, Canada, Europe and the USA. All of them have been, and continue to be, very tough customers. And it was, and is, a pleasure working with all of them. I owe a lot to them: the tougher the customer, greater the satisfaction of winning them over with your work.

The end of 2008 and the beginning of this year has brought more learning opputunities - in XUL, XBL, JQuery, technically challenging ASP.NET projects - and another tough customer.

Looks like 2009 will be a good year too.

Keeping up the net presence

Maintaining a presence on the net has not been easy.

I am on Orkut since June 2007. Till recently I lived with 7 scraps. My son discovered me and scrapped "hey dad....just seven scraps ...thats pretty poor...". Since then - due to greetings for the new year - I crossed 10 scraps..whew...feels like an achievement.

A sister invited me to join Shelfari. Seemed like a good idea - books are something I like. Since then I have been bombarded with mails about how inactive a member I am. A Shelfari member, Dr.M, sent across a 'lets be friends' request. Not knowing any Dr.M's, I rejected his/her hand of friendship, only to find out later that he was a one time customer.

I have been around on Blogger since October 2006. Signing up was easy.

But I have decided to make amends. Blog I shall - sounds like a new year resolution.