When I said that libraries suck, I wasn't being precise.1 Libraries do lots of things well. They allow programmers to quickly prototype new ideas. They allow names to have multiple meanings based on context. They speed up incremental recompiles, they allow programs on a system to share code pages in RAM. Back in the desktop era, they were even units of commerce. All this is good.
What's not good is the expectation they all-too-frequently set with their users: go ahead, use me in production without understanding me. This expectation has ill-effects for both producers and consumers. Authors of libraries prematurely freeze their interfaces in a futile effort to spare their consumers inconvenience. Consumers of libraries have gotten trained to think that they can outsource parts of their craft to others, and that waiting for 'upstream' to fill some gap is better than hacking a solution yourself and risking a fork. Both of these are bad ideas.
To library authors
Interfaces aren't made in one big-bang moment. They evolve. You write code for one use case. Then maybe you find it works in another, and another. This organic process requires a lengthy gestation period.2 When we try to shortcut it, we end up with heavily-used interfaces that will never be fixed, even though everyone knows they are bad.
A prematurely frozen library doesn't just force people to live with it. People react to it by wrapping it in a cleaner interface. But then they prematurely freeze the new interface, and it starts accumulating warts and bolt-on features just like the old one. Now you have two interfaces. Was forking the existing interface really so much worse an alternative? How much smaller might each codebase in the world be without all the combinatorial explosion of interfaces wrapping other interfaces?
Just admit up-front that upgrades are non-trivial. This will help you maintain a sense of ownership for your interfaces, and make you more willing to gradually do away with the bad ideas.
More changes to the interface will put more pressure on your development process. Embrace that pressure. Help users engage with the development process. Focus on making it easier for users to learn about the implementation, the process of filing bugs.
Often the hardest part of filing a bug for your users is figuring out where to file it. What part of the stack is broken? No amount of black-box architecture astronomy will fix this problem for them. The only solution is to help them understand their system, at least in broad strokes. Start with your library.
Encourage users to fork you. "I'm not sure this is a good idea; why don't we create a fork as an A/B test?" is much more welcoming than "Your pull request was rejected." Publicize your forks, tell people about them, watch the conversation around them. They might change your mind.
Watch out for the warm fuzzies triggered by the word 'reuse'. A world of reuse is a world of promiscuity, with pieces of code connecting up wantonly with each other. Division of labor is a relationship not to be gotten into lightly. It requires knowing what guarantees you need, and what guarantees the counterparty provides. And you can't know what guarantees you need from a subsystem you don't understand.
There's a prisoner's dilemma here: libraries that over-promise will seem to get popular faster. But hold firm; these fashions are short-term. Build something that people will use long after Cucumber has been replaced with Zucchini.
To library users
Expect less. Know what libraries you rely on most, and take ownership for them. Take the trouble to understand how they work. Start pushing on their authors to make them easier to understand. Be more willing to hack on libraries to solve your own problems, even if it risks creating forks. If your solutions are not easily accepted upstream, don't be afraid to publish them yourselves. Just set expectations appropriately. If a library is too much trouble to understand, seek alternatives. Things you don't understand are the source of all technical debt. Try to build your own, for just the use-cases you care about. You might end up with something much simpler to maintain, something that fits better in your head.
notes
1. And trying to distinguish between 'abstraction' and 'service' turned out to obfuscate more than it clarified, so I'm going to avoid those words.
2. Perhaps we need a different name for immature libraries (which are now the vast majority of all libraries). That allows users to set expectations about the level of churn in the interface, and frees up library writers to correct earlier missteps. Not enough people leave time for gestating interfaces, perhaps in analogy with how not enough people leave enough time for debugging.
comments
Comments gratefully appreciated. Please send them to me by any method of your choice and I'll include them here.
So you have two choices: backport all bug fixes to every version of your library where you made a breaking change or simply don't make breaking changes.
New ideas take time to percolate through and be acted on. And that's fortunate, because I'm not nearly confident enough about this to claim everybody should start doing this right this instant. Instead I'm experimenting with profligate forking in a little side project of mine.
We already build software with such complex social dependencies that no one developer team can really afford to take ownership of it all. For instance, consider how expensive it has been to "seek alternatives" for entrenched consumer platforms like Windows, Flash, and von Neumann architectures. At some point, we mostly-isolated developers must find places to rest, and API documentation gives us something to believe in. The reassurance we get from this documentation may still be feeble, heavily dependent on social faith, but fortunately we continue to find objective ways to validate it.
As you say, division of labor does require knowing what guarantees our components need and what guarantees external components can be expected to fulfill. However, sometimes we must be willing to impose requirements on systems we don't quite understand, since at least one of the systems we try to interact with is the outside world!
I'm not trying to be purist about this -- especially since we don't understand most of even the software that's technically owned by us. I'm just asking that we think of the entire stack as under our ownership. When you find out about something broken in it, begin first by fixing it in your stack. Then worry about what to do next.
Hey, it just occurred to me that I'm asking for an attitude of Kaizen [1] [2] towards our software stack.
I find it hard to believe in a universal concept of "broken." Even if I trip on a rug, I don't want to pull it out from under someone else. ;)
I'm afraid I have to settle into a culture—get with the program, so to speak—before I'm confident enough to derug it.
It's something so pervasive that we all take it for granted, this idea that we have to put the communal good above our own when it comes to these levels of abstraction. But individualism hath its merits.