Symphonious

I’ve been in New York the past couple of weeks planning out a new project. Part of that work has been working out how the new thing will be built - what protocol and contract changes are needed and how does it all fit in with the existing system? Ultimately we want to create a specification that we have a reasonably high confidence in, but getting that means working through a lot of scenarios and potential designs which is best done on a whiteboard. We found a really powerful way of using AI to help without winding up with a giant pile of AI slop.

Prior to getting together in person, there was a good idea of the customer requirements and a steel-man approach we could use. That was pretty much entirely AI slop, but AI slop is a great way to play around with a PoC and try out many different ideas quickly. From there we shut ourselves in a conference room and did what engineers have been doing forever - we talked through the design, scribbled on a whiteboard and tried to find ways to either improve or break it. What’s relatively new for me (been doing it for most of the year) is to also dial into a Google Meet and have it record a transcription of everything in the room. Prior to AI you’d never read such a transcription - it might be useful to search and find specific decisions but mostly it went unused. With AI, each evening I got it to summarise the transcript and update a notion page that captured our work. More likely to be useful at some point - especially after kicking the AI a few times to make it an up to date record, not a historical summary of all discussions. That page became a pretty good summary of decisions we’d made and where we were up to - still clearly AI slop but usable.

What really surprised me though, was that after a few days of going back and forward on design options, changing our minds, breaking things, fixing them, getting confused ourselves and referencing older designs we had to change, we just ask AI to read the transcripts and write a spec. It did have the benefit of starting from our existing specs repo which gave a large corpus of examples for the style to use. It churned out a quite readable and 90% accurate spec. We still had to go through it all and made quite a lot of corrections but it saved us a day or two of manual writing, and even without AI there’s always a ton of things to fix.

I’ve now read the spec end to end a few times as well as digging into specific parts to help build task break downs and things like that - it reads really well and while there’s a few AI-isms through it, it’s definitely not slop and is really useful. Each milestone in the project still has a task to “write” the relevant piece of the spec to make sure it is updated with any final adjustments we make from actual implementation and to remove the last few AI-isms but it was a significant time saver.

Perhaps even more than the work hours we saved though, is the immediacy of having spec documents ready. I was only available in-person for two weeks and we had a lot to get through - normally we’d have just made some notes or recordings and left writing the actual spec for afterwards since it can be done more async. But then we’d be left to review it async rather than being able to do that in person. Given our first pass review identified a few sections we needed to significantly rework because we’d missed something, the AI generated spec let us resolve that in person with high bandwidth communication. Much better than having to schedule a lot of early morning calls for me.

And of course, afterwards we can have AI generate really bad songs about everything we discussed, full of all kinds of in jokes and weird sayings that happened in the room. But it only really makes sense if you were in the room where it happened…