Sep 24, 2010
How I query Apache logs from the commandline
When I build services or write online I want to get feedback. Did anybody use
them? Read my latest post? When my site has just a little traffic Google
Reader summarizes too much. I want to be able to get my hands dirty with the
data, to be able to drill down to individual user sessions to see how people
interact with my site. How many real users did I have yesterday? Did somebody
link to my latest blog post? How many people clicked on that link on Hacker News? Did any of them stick
around and browse to other pages?
After several attempts at hard-coded scripts to answer such questions, I came
up with a little collection of scripts that can be composed using pipes.
Here's an example session on my commandline:
How many uniques did I get yesterday?
$ cat_logs access.log | dump_field IP | sort | uniq | wc -l
Focus on real human beings.
$ cat_logs access.log | skip_crawlers | dump_field IP | sort | uniq
Wow, 15 IPs? Did they stay long?
$ cat_logs access.log | skip_crawlers | dump_field IP | sort | freq
Hmm, so 4 users browsed several pages. Where are they coming from?
$ cat_logs access.log | skip_crawlers | dump_field REFERER
Ah, they're all coming from http://news.ycombinator.com/item?id=1702108.
So people actually clicked on that comment of mine, even though there were no
votes or responses. Interesting..
This one person viewed 10 pages. What did they see?
$ cat_logs access.log | filter_field IP xxx.xx.xxx.xxx
So they visited twice yesterday, once in the morning and once late at night. And
clicked through to different sites each time.
Read more →
* *
Aug 13, 2010
Advertising vs Spam
You just built something and are trying to get the word out. What are the
ethics of telling a bunch of strangers about it? Is all unsolicited
communication spam? If I send a message to three people, is that bulk? What
if I send a million mails, each email by itself? What if the wording of the
messages is different? How different does it need to be?
Calling @addressed tweets and facebook events “spam” is
increasingly meaningless; let's reserve the word for truly egregious
messages. Instead, if you're considering telling acquaintances or strangers
about something new, this formula may be useful:
Read more →
* *
Apr 4, 2010
Perspectives on happiness
Chuang-Tzu:
Happiness is the absence of the striving for happiness.
Tim
Ferriss: The opposite of happiness is boredom.
Eliezer
Yudkowsky: When people complain about the empty meaningless void, it is
because they have at least one problem that they aren't thinking about
solving — perhaps because they never identified it.
Alex Krupp: Given
perfect freedom people have a tendency to do just enough to make themselves
minimally happy, even if greater happiness is ultimately attainable.
me: There is no 'minimally happy'. Different things either make you
happy or they don't. However, happiness from a source can last a long or
short duration, ebb faster or slower.
Paul Graham: Unproductive
pleasures pall eventually.
Read more →
* *
Apr 2, 2010
Uptake happens fast — or not at all
Mike Speiser:
Most of today’s massive consumer web properties experienced exponential
growth fairly shortly after launch. A few thousand users over a few months is
probably sufficient to find out it you have hit a nerve.
Stephen O'Grady:
Whatever the reasoning, more and more developers, projects and firms were
transitioning away from centralized to decentralized. And happier for it. The
trendline was clear, which is why we weren’t exactly going out on a limb
predicting the ascension of Git, Mercurial and their brethren.
permalink
* *
Feb 20, 2010
Thinking critically about the ideal of a techno-utopia
Technology can compromise resolve. East Germans who watched West
German television were paradoxically more satisfied with life in their
country. The fact that Dresden—where the 1989 protests started—lies too
far and too low to have received Western broadcasts may partly explain the
rebellious spirit of the city's inhabitants. While we fret about the
Internet's contribution to degrading the civic engagement of American kids,
all teenagers in China or Iran are presumed to be committed citizens who use
the Web to acquaint themselves with human rights violations committed by
their governments. For the vast majority of Internet users, increased access
to information is not always liberating. With their endless supply of
entertainment, Twitter and Facebook might make political mobilization harder,
not easier.
Technology empowers all sides equally. We cling to the view that all
non-state power in authoritarian countries is good, while state power is evil
and always leads to suppression. Not all opponents of the Russian or Chinese
or even Egyptian state fit the neoliberal pattern. Nationalism, extremism and
religious fanaticism abound. Facebook and Twitter empower all groups—not
just the pro-Western groups that we like.
Technology drives decentralization;
demonstration requires centralization. Thanks to the decentralization
afforded by the Internet, Iran's Green Movement couldn't collect itself on
the eve of the 31st anniversary of the Islamic revolution. It simply drowned
in its own tweets.
Technology increases noise and misinformation. We assume the Internet
makes it easy for citizens to see who else is opposing a regime and then act
collectively based on that shared knowledge. In the age of the Spinternet,
cheap online propaganda can easily be bought with the help of pro-government
bloggers. Add to that the growing surveillance capacity of modern
authoritarian states—greatly boosted by information collected through
social media.
Technology shines a harsh light. Diplomacy is, perhaps, one element of
the U.S. government that should not be subject to the demands of "open
government"; whenever it works, it is usually because it is done behind
closed doors.
—Paraphrasing Evgeny Morozov
permalink
* *
Jan 23, 2010
Tyranny of the majority, or regulatory capture? Just be more agile.
Plato, de
Tocqueville, et al.: In a democracy, the greatest concern is that the
majority will tyrannize and exploit diverse smaller interests.
Mancur Olson:
Narrow and well-organized
minorities are more likely to assert their interests over those of the
majority.
Neil Freeman: Just redistrict
the states after each census.[1]
me: Can this idea be generalized? Minorities can be oppressed or
powerful; strive to so intertwine motivations that minorities are eliminated.
Track minority power and standard deviation of group size as a
quality metric for democracy.
But maintain diversity.
And don't allow collusion
to foster bubbles.
Read more →
* *
Dec 20, 2009
Books can be of any length — if they're mysteries
“People apparently only read mystery stories
of any length. With mysteries, the longer the better and people will read any
damn thing. But the indulgent, 800-page books like The Brothers
Karamazov or Moby-Dick of a hundred years ago are just not going
to be read anymore.”
— Cormac McCarthy. Contrast Jeff Bezos.
permalink
* *
Dec 16, 2009
“Better get busy”
“Frank Curiel Field in Curaçao looks like every small-town baseball
field you've ever seen. But it's filled with primal cues, a window through
which kids can see the ascending realms of heaven stacked above them in neat
levels. First comes making the league all-star team (one of those
guys). Then playing for Williamsport (those guys). Then just above that
is getting signed by a scout, playing in the major leagues (being one of
those guys). For a six-year old at this field, these aren't gauzy
dreams; they are tangible steps on a primal ladder of selection reflected in
the crackle of the radio, the clutter of the trophies, the glint of the
scout's sunglasses. It is sort of like standing in the Sistine Chapel. The
proof of paradise is right here: all you have to do is open your eyes.”
—Daniel Coyle. Curaçao has been to six of the last eight semifinals at the Little League World Series.
permalink
* *
Dec 11, 2009
Search wars
“Search is broken like browsers were broken in 2002.”
— Asa Dotzler of Firefox doesn't like Google
Read more →
* *