Trench Warfare Problems

I've been working through my research lately, and I've been fantasizing about how sublime it must have been to work through this research when the field was young. At times it feels awfully like trench warfare–lots of effort for little gain–and I've been going down different rabbit holes as a result of how stagnant my research has been feeling.

This idea that the problems now are harder to solve isn't new either. Contemporary physicists have long elided this problem of tedious trench warfare by splitting into sides: the string theorists, the einstein gravity people, and even when physicists don't divide themselves, worshipping at the altar of mathematics unites them. Moore's law is dying (or dead), previous innovators like Intel consistently screw up their transitions to <10nm while Apple eats their lunch, and software innovations like deep learning exist at the mercy of the hardware lottery–the coincidental matching of hardware and software determines the success of an idea, not the effectiveness of the idea itself. From the looks of it, the low hanging fruit has all been picked, and the problems are now exponentially harder (and therefore exponentially more expensive) to solve, leading to a feast-or-famine scenario between large, established players with huge war chests and small and nimble players hoping to exit with a buyout or become a large player on their own.

This applies to the job market as well. Dan Luu posits that developer compensation has become bimodal–split between the developers who occupy the FAANG-tier companies and those who exist in "normal" companies. Similar to lawyers, the higher compensated band begins to exhibit signs of credentialism, lawyers at major law firms rely on previous work experience and law school name, while software developers often tout their experience as an "ex-FAANG" programmer. This has led to a career moat of sorts: this type of credentialism makes it very difficult to break into the higher compensation band without having the social network to refer you in, which is why referrals at these companies (that allow you to bypass the muddled resume screen) are so desired.

In some ways, this is emblematic of our loss of clear, actionable information. James Bridle suggests that we're entering a new dark age, not because information is difficult to find, but rather because there's too much of it, and the data we have aren't good predictors of the future. The world is becoming more unstable at the same time we (both our computers and ourselves) explode with information. It's far easier to generate knowledge than interpret it, and it's even more difficult to validate. Academia is rife with this, we've prized knowledge production and accumulation while eschewing replication, like dragons who have acquired large golden coin hoards, only to find out later they are made of candy.

This is the essence of trench warfare problems: nothing is simple anymore. The proliferation of complexity means every gain is marginal, the battle lines shift only inches in the dirt with every piece of knowledge produced, consumed, or validated.

There's some benefits to this as well though. Because trenches don't move far, introductory level knowledge on any topic is extremely accessible. Each new researcher going to the trenches follows well defined paths, and the explosion of online guides, blog posting, and moocs are a autodiadect's dream. Even business use cases have sprung up around this knowledge proliferation: discoverability and learning to rank are among the most desired machine learning models today. For those wanting to get into a variety of different fields, there's no better time.