What we want out of software
The best software engineers are endlessly curious about how things work in a way that can bewilder other brain orientations.
Once upon a time I attended a DEFCON talk given by an engineer who found three old wireless routers of unknown provenance, and he decided he wanted to figure out how they worked. He could tell the chips had 14 16-bit registers (similar to an 8086 chip), and he was able to hook up some circuitry to try different states. Eventually he just brute-forced a ton of different states to see how things changed: 2^224 different states, or more states than there are grains of sand on earth. He then backed into an instruction set, and now he understands those chips.
Was there any practical reason to do that? Absolutely not. His innate curiosity pushed him to do it. I'll submit that innate curiosity, continually pushing to deeply understand some hidden system and its ins and outs, is a massive value-booster for any enterprise.
Lots of human beings aren't wired to tinker with technology in that way. And that affects how software companies go to market.
ServiceTitan is a software company that serves the trade industry: HVAC specialists, plumbers, and electricians. They do everything from invoicing to customer relationship management in a single platform tailored to that industry. It's software built for non-software people - people who would rather be out in the field working with customers doing their jobs instead of tracking invoices for the accountants.
ServiceTitan made $700M in revenue last year across ~8000 customers: so customers are very happy to continue paying ticket prices nearly $100K on average to make this work. Their founders helped their parents run their own trade businesses and they deeply know the industry.
Despite something that looks like a perfect product, ServiceTitan has a ton of professional services revenue that is delivered at a loss. Their perfect product built for a specific industry for people who know that industry well still requires human professional services in order to work.
When I was at Recharge, we also delivered various professional services at a loss. We were also a software company purpose-built for a specific industry. Yet we also consistently ran into people who wanted to run their coffee business or their soap business and didn't want to tinker with software. They wanted to get a human being on the phone to do it for them. The improved time-to-outcome was the goal: not the software.
Consider the engineer in my opening story. The outcome was the knowledge. He wasn't trying to get WiFi up and running with the ultimate end of watching Netflix faster. His outcome was understanding how this tool worked.
So many organizations are properly grappling with AI because this stubborn problem insistently won't go away, and AI promises that maybe this time it could. A company like ServiceTitan or Recharge won't need to employ any of those pesky human beings doing professional services - AI will just know how to help, and it'll be cheaper! It'll get me what I want immediately! It'll surface all the right information, and only the right information, exactly when I want it!
Then the question inevitably comes up: "ok, well, what do you want?"
And the truth is that we don't know, not until we talk to some other humans. We're social creatures. We derive our ideas and positions through feedback from others. AI gives us a facsimile of that which works up until the point that we need to go communicate instructions with someone else. Then we have to uncover all the pros, cons, tradeoffs, and issues that pop up once we need to chart a course and go execute.
Computer science has a term for this type of problem: it's NP-hard. It cannot be completed in polynomial time. The more variables you add to the problem, it gets exponentially harder to solve. You either have to brute-force the whole thing or pick an optimization that is "pretty good" and just go for it.
The canonical example of an NP-hard problem is the traveling salesman problem. If you have a person who needs to visit N different cities with different travel times between them, what is the optimal route to visit all cities? To find a perfect solution to the problem, you have to brute-force it. Change one variable about the problem and you have to brute-force it again. Yet we can easily take shortcuts to get to "pretty good" and move on with our lives.
"What humans want" is an NP-hard problem. It's impossible to be perfect. We can get to "pretty good" through optimizations, and a lot of those optimizations require other human beings. AI can help get in the right direction, but it's not cracking the problem yet - and because of the computer science around NP-hard problems, I'll submit that it never fully will.