What I learned while trying to build a production-ready nearest neighbor system