The fairest way to judge an AI tool in an HVAC company is a 30-day pilot measured against your own baseline, with every cost counted, including the staff time spent reviewing what the AI did. This guide gives the baseline numbers to pull, a worked example you can redo with your own figures, the weekly scorecard we use, and the rules for deciding at the end. You can run it with any vendor.
Step 1: pull your baseline before anyone touches a tool
You can't measure improvement without a starting point, and most HVAC offices don't have one written down. Pull these numbers for the last full month, and the same month last year if you have it:
| Area | Baseline numbers |
|---|---|
| Calls | Total inbound, answered live, voicemail, abandoned, after-hours |
| Bookings | Service calls booked, maintenance visits booked, replacement estimates booked |
| Quotes | Estimates sent, followed up within 7 days, closed, average value |
| Admin | Minutes per job on notes and data entry (time ten jobs) |
| Experience | Complaints, negative reviews mentioning phones or communication |
Your phone system or carrier portal usually shows call counts. If it doesn't, count voicemails and missed-call notifications for two weeks. Rough is fine; missing is not.
Step 2: define one job and one success number
A pilot fails when it tries to prove everything at once. Pick one AI task, and one number that decides success:
- AI call answering: extra calls booked or callback-confirmed per month.
- Estimate follow-up: close rate on quotes above a set value.
- Dispatch and reporting helpers: office minutes saved per job.
- Content assistance: finished, fact-checked pages per month.
Write it down with a target before the pilot starts. "We keep it if it books at least 12 extra jobs in 30 days" is a real decision rule. "Let's see how it goes" is not.
Step 3: do the cost maths honestly
Here is a worked example for overflow call answering. Replace every number with your own.
Assumptions (illustrative only):
- Missed calls last January: 70
- Share that were genuine service or sales inquiries: 60%, so 42
- AI answers 80% of overflow in the pilot: about 34 genuine callers reached
- Of those, 40% book or confirm a callback that books: about 13 jobs
- Average gross margin per booked job, after parts and labour: $180
Gain: 13 × $180 = $2,340 in gross margin for the month.
Costs (illustrative):
- Tool subscription and usage: you get this from the vendor's official pricing page
- Staff review time: 20 minutes a day for 22 working days, about 7 hours, at $30 an hour = $210
- Setup time spread over a year: whatever your setup took, divided by 12
If the tool and setup together cost less than about $2,100 a month in this example, the pilot pays. Notice how sensitive the result is to the booking rate: at 20% instead of 40%, the gain halves to about $1,170, and many tools would no longer pay in a mild month.
Run the same sum for a busy month and a quiet month. Many HVAC companies find AI answering pays clearly in peak heating and cooling seasons and only marginally in spring and autumn. That is a legitimate result, and it may mean running the AI only on overflow during peaks.
Step 4: the weekly scorecard
Every Friday during the pilot, fill in one row. It takes ten minutes.
| Week | Success number | Errors found | Escalations | Hang-ups or complaints | Review time | Notes |
|---|---|---|---|---|---|---|
| 1 | ||||||
| 2 | ||||||
| 3 | ||||||
| 4 |
Errors found are things your staff caught: wrong addresses, a missed emergency, an invented answer. Escalations are handoffs to a person; very few can mean the AI is keeping calls it should pass on. Hang-ups or complaints tell you how customers feel. Review time keeps the cost side honest.
What healthy looks like
- Errors fall each week as scripts are fixed.
- Review time falls but never hits zero.
- Complaints stay at or below baseline.
- The success number trends toward the target by week three.
Warning signs
- The same error repeats after a fix: the tool may not be capable of the task.
- Staff stop reviewing because "it's fine": that is when problems slip through.
- Callers repeatedly ask for a human early in the call: the greeting or disclosure needs work, or the AI should be after-hours only.
- Any safety-related mistake: pause the AI immediately and fix it before restarting.
Step 5: decide, in writing
At day 30, hold a 30-minute meeting with the owner and the people who reviewed the output. Choose one of three outcomes:
- Keep: the success number met its target, costs are covered, errors are under control. Set a monthly review and a smaller weekly sample.
- Change and rerun: close to target, with clear fixable problems. Run another 30 days with specific changes.
- Stop: the target was missed by a wide margin, or errors or complaints didn't improve. Cancel the tool and keep the lessons.
Stopping is a good outcome when the numbers say so, and it costs far less than a year of paying for a tool that quietly annoys customers. The worst result is a tool that keeps running for a year because nobody measured it.
Common pilot mistakes
- Changing two things at once. Launching AI answering in the same month as a new ad campaign makes the results unreadable.
- No privacy setup. Recording notices, consent records and data settings belong in week zero. See our consent and privacy guide.
- Testing on real customers first. Scripted test calls come before live traffic; our phone failure guide has a test plan.
- Ignoring seasonality. Compare like months.
- Letting the vendor run the scorecard. Your numbers, your spreadsheet.
Where marketing fits
AI handles the leads you already get; it doesn't create demand. If your call volume is low, the bigger win may be marketing. For context from our marketing work, CombiBoiler.ca reached 300+ monthly leads and a 300% ROI, and those volumes are where AI answering and follow-up start to matter. Our services and pricing pages explain how the two connect.
Frequently asked questions
Why 30 days?
Long enough to see a range of days and some volume swings, short enough that you aren't locked into a tool that isn't working. In shoulder seasons you may need longer to get enough calls for a fair read.
What if the pilot falls in a quiet month?
Compare like with like: this October against last October, not against January. If you have no history, run the AI on half the overflow calls and compare the two halves.
Who should review AI output during the pilot?
The person who would have done the work: your CSR for calls, your comfort advisor for quote follow-ups, your dispatcher for summaries. They spot errors fastest and their buy-in decides whether the tool survives.
What is a fair cost to include?
The tool subscription and usage, phone or texting charges, setup time, and staff review time valued at their hourly cost. Leaving out review time makes almost any AI look profitable.