Sep 24, 2010
How I query Apache logs from the commandline

When I build services or write online I want to get feedback. Did anybody use them? Read my latest post? When my site has just a little traffic Google Reader summarizes too much. I want to be able to get my hands dirty with the data, to be able to drill down to individual user sessions to see how people interact with my site. How many real users did I have yesterday? Did somebody link to my latest blog post? How many people clicked on that link on Hacker News? Did any of them stick around and browse to other pages?

After several attempts at hard-coded scripts to answer such questions, I came up with a little collection of scripts that can be composed using pipes. Here's an example session on my commandline:

How many uniques did I get yesterday?

$ cat_logs access.log | dump_field IP | sort | uniq | wc -l

Focus on real human beings.

$ cat_logs access.log | skip_crawlers | dump_field IP | sort | uniq

Wow, 15 IPs? Did they stay long?

$ cat_logs access.log | skip_crawlers | dump_field IP | sort | freq

Hmm, so 4 users browsed several pages. Where are they coming from?

$ cat_logs access.log | skip_crawlers | dump_field REFERER

Ah, they're all coming from http://news.ycombinator.com/item?id=1702108. So people actually clicked on that comment of mine, even though there were no votes or responses. Interesting..

This one person viewed 10 pages. What did they see?

$ cat_logs access.log | filter_field IP xxx.xx.xxx.xxx

So they visited twice yesterday, once in the morning and once late at night. And clicked through to different sites each time.

Read more →
* *
Aug 13, 2010
Advertising vs Spam

You just built something and are trying to get the word out. What are the ethics of telling a bunch of strangers about it? Is all unsolicited communication spam? If I send a message to three people, is that bulk? What if I send a million mails, each email by itself? What if the wording of the messages is different? How different does it need to be?

Calling @addressed tweets and facebook events “spam” is increasingly meaningless; let's reserve the word for truly egregious messages. Instead, if you're considering telling acquaintances or strangers about something new, this formula may be useful: Read more →

* *
Apr 4, 2010
Perspectives on happiness

Chuang-Tzu: Happiness is the absence of the striving for happiness.

Tim Ferriss: The opposite of happiness is boredom.

Eliezer Yudkowsky: When people complain about the empty meaningless void, it is because they have at least one problem that they aren't thinking about solving — perhaps because they never identified it.

Alex Krupp: Given perfect freedom people have a tendency to do just enough to make themselves minimally happy, even if greater happiness is ultimately attainable.

me: There is no 'minimally happy'. Different things either make you happy or they don't. However, happiness from a source can last a long or short duration, ebb faster or slower.

Paul Graham: Unproductive pleasures pall eventually. Read more →

* *
Apr 2, 2010
Uptake happens fast — or not at all

Mike Speiser: Most of today’s massive consumer web properties experienced exponential growth fairly shortly after launch. A few thousand users over a few months is probably sufficient to find out it you have hit a nerve.

Stephen O'Grady: Whatever the reasoning, more and more developers, projects and firms were transitioning away from centralized to decentralized. And happier for it. The trendline was clear, which is why we weren’t exactly going out on a limb predicting the ascension of Git, Mercurial and their brethren.

permalink

* *
Feb 20, 2010
Thinking critically about the ideal of a techno-utopia

Technology can compromise resolve. East Germans who watched West German television were paradoxically more satisfied with life in their country. The fact that Dresden—where the 1989 protests started—lies too far and too low to have received Western broadcasts may partly explain the rebellious spirit of the city's inhabitants. While we fret about the Internet's contribution to degrading the civic engagement of American kids, all teenagers in China or Iran are presumed to be committed citizens who use the Web to acquaint themselves with human rights violations committed by their governments. For the vast majority of Internet users, increased access to information is not always liberating. With their endless supply of entertainment, Twitter and Facebook might make political mobilization harder, not easier.

Technology empowers all sides equally. We cling to the view that all non-state power in authoritarian countries is good, while state power is evil and always leads to suppression. Not all opponents of the Russian or Chinese or even Egyptian state fit the neoliberal pattern. Nationalism, extremism and religious fanaticism abound. Facebook and Twitter empower all groups—not just the pro-Western groups that we like.

Technology drives decentralization; demonstration requires centralization. Thanks to the decentralization afforded by the Internet, Iran's Green Movement couldn't collect itself on the eve of the 31st anniversary of the Islamic revolution. It simply drowned in its own tweets.

Technology increases noise and misinformation. We assume the Internet makes it easy for citizens to see who else is opposing a regime and then act collectively based on that shared knowledge. In the age of the Spinternet, cheap online propaganda can easily be bought with the help of pro-government bloggers. Add to that the growing surveillance capacity of modern authoritarian states—greatly boosted by information collected through social media.

Technology shines a harsh light. Diplomacy is, perhaps, one element of the U.S. government that should not be subject to the demands of "open government"; whenever it works, it is usually because it is done behind closed doors.

Paraphrasing Evgeny Morozov

permalink

* *
Jan 23, 2010
Tyranny of the majority, or regulatory capture? Just be more agile.

Plato, de Tocqueville, et al.: In a democracy, the greatest concern is that the majority will tyrannize and exploit diverse smaller interests.

Mancur Olson: Narrow and well-organized minorities are more likely to assert their interests over those of the majority.

Neil Freeman: Just redistrict the states after each census.[1]

me: Can this idea be generalized? Minorities can be oppressed or powerful; strive to so intertwine motivations that minorities are eliminated. Track minority power and standard deviation of group size as a quality metric for democracy.

But maintain diversity. And don't allow collusion to foster bubbles. Read more →

* *
Dec 20, 2009
Books can be of any length — if they're mysteries

People apparently only read mystery stories of any length. With mysteries, the longer the better and people will read any damn thing. But the indulgent, 800-page books like The Brothers Karamazov or Moby-Dick of a hundred years ago are just not going to be read anymore.”

Cormac McCarthy. Contrast Jeff Bezos.

permalink

* *
Dec 16, 2009
“Better get busy”

Frank Curiel Field in Curaçao looks like every small-town baseball field you've ever seen. But it's filled with primal cues, a window through which kids can see the ascending realms of heaven stacked above them in neat levels. First comes making the league all-star team (one of those guys). Then playing for Williamsport (those guys). Then just above that is getting signed by a scout, playing in the major leagues (being one of those guys). For a six-year old at this field, these aren't gauzy dreams; they are tangible steps on a primal ladder of selection reflected in the crackle of the radio, the clutter of the trophies, the glint of the scout's sunglasses. It is sort of like standing in the Sistine Chapel. The proof of paradise is right here: all you have to do is open your eyes.”

Daniel Coyle. Curaçao has been to six of the last eight semifinals at the Little League World Series.

permalink

* *
Dec 11, 2009
Search wars

Search is broken like browsers were broken in 2002.”

Asa Dotzler of Firefox doesn't like Google Read more →

* *
Dec 3, 2009
Sniper Business

https://www.youtube.com/watch?v=sfz2D6dglCo

via Scott Locklin

permalink

* *
archive
projects
writings
videos
subscribe
Mastodon
RSS (?)
twtxt (?)
Station (?)