At Downtobid, I work on the AI that powers our AI-construction plans reader. We did not get this product right on the first try. In fact, we made many attempts tweaking the AI and product output. Every 2-3 months, I would update the /estimator community on reddit.

Our vision

We’re building the world’s first AI-powered construction spreadsheet. GCs will be able to upload their plans and AI will instantly perform the following actions.

  • Identify scopes of work
  • Generate a sublist matched against these scopes of work
  • Generate a planroom
  • Generate ITBs personalized to every sub
  • Generate line items for each scope of work
  • Organize this into a modern construction-specific spreadsheet that offers collaboration, bid-leveling features
DTB V1

Evolution of the product

We started building this product on Oct 2023 (1-year ago). At the time, it was clear that the AI-models (GPT-3) were not ready for this task, but AI would eventually advance to a state where this would be possible. The design challenge is … “how do you build something useful that will also set you up to build the ultimate AI-powered construction spreadsheet?” Here are some of the lessons we learned along the way.

V1 - AI to understand structure

Initially, the naive approach is to upload a PDF and simply chat with it using your LLM. Typically, this is powered through OCR and a RAG-type system. The problem is that construction plans have a lot of semantic meaning that cannot be captured through OCR (table, specs, structures, drawings, floor plans). The first problem we had to solve was to build an AI that could understand the “structure” of construction plans. This is done via computer vision and segmentation.

DTB V2

V2 & V3 & V4 - AI to solve citations

After we nailed structure, we started generating line items, which is the holy-grail in preconstruction. If you can do this accurately, you make every single construction estimator 10x more productive. The line-item generation problem is framed as “given a set of construction plans, can you generate a list of items that will be a potential cost on the project?”

It turns out this is hard for 2 reasons. First, LLMs cannot reason about construction so they often make up bogus line items that aren’t accurate to the construction drawings. Second, estimators naturally do not trust LLMs, so even if an LLM generates correct line items, estimators need to see evidence on the original plans to believe what the LLM generated. Our strategy was to first solve for citations, which means building UI that makes it dead-easy for an estimator to validate the LLM’s interpretation of line-items. We picked this strategy because we knew that LLMs would not be 100% accurate on line-items for a very very long-time. Therefore, without 100% accuracy, you need the ability to validate. Moreover if you can validate quickly, then you have a useful product.

Inspirations (NotebookLM & Perplexity)

Google’s NotebookLM and Perplexity were good exemplars for incorporation citations into their product. The insight is they enable the user to to verify the sources the LLM used to generate the response.

Notebook LM Citations
DTB Reader V3

DTB citations

In version 1, we provided a list of line-items and a list of pages that the LLM used to generate the response. This wasn’t good enough.

Citations V1

In version 2, we the line-item to the source page, but estimators still needed to scan the page. Better, but not quite there.

Citations V2

In version 3, we provided a simple citations button so that when an estimator clicked on the line item, they would be taken to the source page, with the evidence highlighted. Now we’re starting to get pretty good.

Citations V3

We haven’t built version 5 yet, but that version of the construction plans reader now shift from citations UI improvements to solving accuracy for the line item generation problem.

Reddit posts

Every 2-3 months, I post an update on the product & technology to the /estimator community on reddit. It’s interesting to see how the technology evolves based on feedback. Even more interesting is to see how the reactions to the product go from skeptical to believable.