Why I don't document quirks of a feature on the feature itself.

Every product manager knows this situation:

• A user works with feature X1.
• They find a limitation / bug / quirk and want to work around it.
• The perfect workaround or alternative is feature X2, but without knowing that X2 exists, the user doesn’t find it and spends a lot of time looking for it.
• The user requests X2 be documented on X1, because that would have saved them a ton of time.

This is such a common pattern, and while it’s perfectly understandable for such a user to request this, it is so terribly wrong to give in to this user’s request. Why is it wrong?

The features are unrelated

Most of the time, the two features X1 and X2 are unrelated and just “happen” to be connected in this specific user’s situation. Let’s take jOOQ, for example. Imagine you work with the SUM(X) aggregate function. For historic reasons, this function (like many other arithmetic functions) returns BigDecimal in jOOQ, so it won’t overflow, irrespective of the input type. The signature of the function is:

public static AggregateFunction<BigDecimal> sum(
    Field<? extends Number> field
) { ... }

This may or may not be a good thing. After all, people are used to generics in jOOQ’s API, so they may have preferred this signature instead (which risks overflowing for types like TINYINT / Byte):

public static <T extends Number> AggregateFunction<T> sum(
    Field<T> field
) { ... }

So, feature X1 is SUM(X). The “quirk” is that it may return an unexpected type. (The quirk is controversial. The relevant issue to address this isn’t very popular, so it may just as well be fine).

Now, for some users who prefer the alternative signature, feature X2 is the coerce() function, a jOOQ function that works like a SQL cast(), but does not impact generated SQL. It’s implemented entirely in the client. Though, cast() would also work here.

For example, to get an Integer sum:

// Instead of:
Record1<BigDecimal> r1 = ctx
    .select(sum(TABLE.COLUMN))
    .from(TABLE)
    .fetchOne();

// Get this (where SQLDataType.INTEGER is used):
Record1<Integer> r2 = ctx
    .select(sum(TABLE.COLUMN).coerce(INTEGER))
    .from(TABLE)
    .fetchOne();    

coerce() is such a useful feature X2 in jOOQ, it’s probably worth knowing about it if you’re a jOOQ power user. It isn’t related to X1 (the SUM(X) aggregate function), although it helps work around a “problem” (or quirk / historic API design choice / etc.) that is specific to SUM(X) (and dozens of other arithmetic functions).

The feature is unknown

But the problem a user might have is this:

• Users have no problem finding SUM(X), the X1. Every supported SQL function has a corresponding jOOQ method in DSL, and jOOQ users are quickly trained to look there for anything they already know from SQL.
• Users do have a problem finding out how to coerce() or cast() a data type, first because 1) they may not know (or have forgotten, or are unable to connect) the concept, and 2) it’s simply not related to the user’s task at hand.

So, a lot of time may be spent looking for a solution. Without knowing what the best solution is, alternatives may be explored. In the worst case, those alternatives also cause (unrelated!) issues, and the even more frustrating XY problem ensues.

Even without the XY problem, a lot of time may be wasted looking for the solution X2.

The urge to document

If the user eventually finds out about X2 (the coerce() function), they might request that X2 be documented on X1, because that would have really helped them. And there’s no doubt that this kind of documentation would have helped this particular user.

BUT!

Most users who work with SUM(X) do not need to know anything about coerce(). They’re perfectly happy with BigDecimal, or alternatively, they may have found an entirely different approach to turning BigDecimal into Integer (e.g. BigDecimal::intValue) and they’re perfectly happy with their solution X3.

If X2 was documented on X1, for most users, that would have just been noise. Perhaps it’s not terrible noise but:

• It’s a first bit of noise on X1 about an unrelated feature X2.
• The next user will find X2 to be insufficient for their problem and would want to have X3 documented as well.
• The next user will find an entirely different limitation in X1 and will want to have their workaround X4 documented as well (for example, the fact that SUM(X) returns NULL instead of the identity value 0 for empty input sets, a common SQL “quirk”).
• The next user will have a limitation on X5 and will want to have X1 and X6 documented there.

The set of features X = {X1, X2, ..., Xn} is large, and if a product is designed well and has a lot of highly reusable components, like jOOQ, then in principle, almost every feature can interact with every other feature.

This is already hard enough to test, but if every combination also has to be documented along with a list of quirks, caveats, gotchas, workarounds, etc. then we’ll never stop documenting things that 99% of users will never need to know and thus perceive as noise.

As an extreme example: Imagine a user who has to pass a List<T> to method(List<T>). But they’re unaware of ArrayList. Not only do they request ArrayList be documented on method(), but also the fact that an ArrayList can be created using new. It’s totally unrelated noise!

There would be so much noise that our user who wanted X2 (the coerce() function) to be documented on X1 (the SUM(X) aggregate function) wouldn’t even find this bit of information in the endless list of quirks. Needless to say that moving that particular quirk to the top of the list of quirk isn’t going to be the right solution to finding this bit of information.

So, what is the best way to help this user?

Personally, I like Stack Overflow. People perceive the moderation as harsh, but the big benefit of this moderation is that good questions are always very specific and thus useful for the next user who finds the question on Google. It’s perfectly fine to ask about the combination of X1 and X2 on Stack Overflow. While the features are unrelated, the combination does help this particular user. So if this user asks:

I want to use X1, but I ran into this quirk. How can I work around it?

Then I could answer:

Use X2 as a workaround. Here’s how: [ … ]

This is much much better than documentation on the Javadoc or in the manual:

• It doesn’t generate noise. 99% of jOOQ users aren’t concerned with this Q&A, so they will never Google it, and thus never encounter this question. So, they’re happy.
• It’s highly specific. The 1% of jOOQ users who do have the same question will find this answer on Stack Overflow (or ChatGPT & co, which plagiarise Stack Overflow), so they’re happy as well.

Conclusion

Documenting quirks is a slippery slope. As a product grows and matures, it accumulates quirks. There are no exceptions to this. Documenting all of these quirks somewhere is important, because users who run into them will want answers.

Documenting these quirks on the feature documentation itself isn’t the right choice, however, because most users won’t be concerned with the quirk. The list of documented quirks will quickly become overwhelming.

A Q&A of some form is usually the best approach, be it:

• A question on Stack Overflow
• An issue on an issue tracker with a related discussion about the quirk and all the possible workarounds