Risks of Employees Using ChatGPT for Financial Tasks

Oct 21

Your accountant is pasting payroll into ChatGPT right now.

Not maliciously. Not because they don't care. Because the numbers don't tie out, it's 8:47 on a Tuesday night, and ChatGPT is three seconds away from maybe giving them an answer before they close the laptop.

This is the conversation most founders haven't had yet, and it's the one I want to have today.

Here's What's Actually Happening

I'll give you three numbers, and then we can move on.

Cyberhaven Labs tracked actual AI usage across seven million workers in 2025. They found that 34.8% of what employees put into AI tools is sensitive corporate data. Two years ago that number was 10.7%. It tripled.

IBM’s 2025 Cost of a Data Breach Report says one in five organizations had a breach linked to shadow AI.

Those breaches cost, on average, $670,000 more than breaches at companies without shadow AI. 97% of companies that experienced an AI-related incident admitted they had no AI access controls in place.

LayerX Security's 2025 research: 77% of employees paste company data into generative AI tools, most of them from personal accounts nobody at the company can see.

That's the air we're breathing. Your team is using ChatGPT. The question is whether you know what they're doing with it.

The Four Stories I Hear Every Month

These are real. Lightly disguised, because the founders who told me them didn't want their names attached, but every one of these happened inside a SaaS startup in the last six months.

The senior accountant. She's trying to find a mismatch between gross payroll and the journal entry that landed in Xero. It's late.

She exports the full payroll register, pastes it into ChatGPT, and asks where the variance is. ChatGPT finds it in thirty seconds.

So does OpenAI's logging system. Names, titles, comp, one-off bonuses for everyone on the team. All of it on a system that the company has no data handling agreement governing.

The ops manager. A vendor renewal is coming up. He uploads the full contract to get a clean summary for the call. The contract has pricing, IP terms, a mutual NDA, and a penalty clause. None of it was ever supposed to leave the company. The summary was great.

The founder. A founder runs a back-of-envelope runway model in ChatGPT by sharing the current bank balance, burn, MRR, and hiring plan. It's exactly the package a social engineer would want. And it was shared with a service that wasn't evaluated for the company's security requirements.

The bookkeeper. A messy list of 40 expense line items, half of them ambiguous. She asks ChatGPT to categorize them. The output looks clean. She imports it.

Two months later, three transactions turn out to be in the wrong accounts, one of which is the kind of misclassification an auditor asks about. The AI was confident. The AI was wrong. Nobody checked.

None of these people did anything wrong, exactly. They did something unsupervised. There's a difference, and the difference is where your exposure lives.

Why Consumer ChatGPT Is a Different Product Than You Think

This is the part most founders have never had explained to them, so I'll be direct.

ChatGPT Enterprise, Business, Edu, Team, and the API do not train on your inputs. They're covered under OpenAI's SOC 2 Type II report. Admins control data retention, the API supports Zero Data Retention for qualifying orgs, and encryption works the way you'd expect.

ChatGPT Free and ChatGPT Plus are different products. Not the same scope.

Not the same retention controls. Not the same contractual posture. It's the product your team is using on their personal laptops with their personal email addresses, and the terms of service are what you'd expect from a consumer app, because it is one.

Most of your accounting risk lives in that gap.

A company running ChatGPT Enterprise with sensible policies is probably fine. A company whose controller logs into chatgpt.com with her Gmail address and pastes in the trial balance is not fine, and nobody at the company has told her why.

Here's the kicker most people don't know. In May 2025, a federal magistrate in the New York Times v. OpenAI case ordered OpenAI to preserve all ChatGPT output logs that would have otherwise been deleted, including free, Plus, Pro, Team, and non-ZDR API traffic. Enterprise and Edu were excluded.

That order was narrowed by September, but not before OpenAI was compelled to produce 20 million de-identified chat logs. If your team's prompts touched consumer ChatGPT during that window, they may still exist on OpenAI's systems today, regardless of what your "delete my data" settings told you at the time.

That's the part I want founders thinking about. The data you think is gone may not be.

The Risks, in Plain English

What are the real risks when employees use ChatGPT for accounting work?

Five things, in order of how often they bite startups.

One. Data you didn't mean to share is now somewhere you don't control. Payroll, customer lists, investor decks, vendor contracts, board packets. All of it can travel through a prompt in seconds. Once it's out, it's out. You can't pull it back.

Two. The output looks right even when it isn't. ChatGPT writes with the same confidence whether it's correct or inventing. That's the whole problem in a financial context. A made-up tax threshold, a misremembered revenue recognition rule, a hallucinated GAAP principle, all of it arrives in the same polished voice. If that language ends up in a board memo or a journal entry narrative, you have an error that's almost impossible to trace.

Three. You may be breaching someone else's confidentiality. Your vendor's pricing. Your customer's contract. Your candidate's offer letter. When that goes into ChatGPT, you may be violating an NDA or a confidentiality clause you forgot you signed. Nobody sent the memo. Nobody had to.

Four. Your compliance story stops making sense. If you handle customer PII, you have SOC 2, GDPR, CCPA, maybe HIPAA obligations about who can process what. Consumer ChatGPT is almost never on the approved sub-processor list. The Enterprise tier can be. Most audits will want to know which one your team is using, and "both, we think" is the wrong answer.

Five. Two accountants, two answers, no reconciliation. Different team members using different prompts produce different categorizations, different narratives, different interpretations. Without governance, you get drift. Drift shows up during Series A or Series B diligence, and by then, you're cleaning up in a hurry.

The Part Nobody Says Out Loud

Shadow ChatGPT use in accounting is not an employee problem. It's an infrastructure problem wearing an employee costume.

Let me say that again, because it's the whole point of this piece.

If your controller is pasting Stripe exports into ChatGPT to reconcile against Xero, your Stripe-to-Xero integration is broken. If your AP lead is using ChatGPT to categorize invoices, your document processing stack (Dext, Hubdoc, whatever) is incomplete. If your ops manager is asking an AI tool to summarize vendor contracts, you don't have a contract system that the team trusts. If your bookkeeper is dumping expense lists into ChatGPT, Ramp or Brex isn't fully configured against your chart of accounts.

Your team isn't routing around your systems because they're sneaky.

They're routing around your systems because the systems don't work yet.

That reframe matters because once you see it this way, the fix is obvious. You don't write a policy memo. You fix the plumbing. Close the workflow gaps, automate the repetitive stuff through ai workflows that actually run, and then give your team approved tools for the work that genuinely needs AI judgment. The shadow use shrinks on its own.

What Actually Works

How should a SaaS startup manage employee ChatGPT use for financial work?

This is the playbook I walk founders through. Six moves. Not a policy manual. A playbook.

One. Ask your team what they're doing. Not in a legal deposition. Over coffee, in Slack, in a short team survey. "What AI tools are you using? What do you use them for? What do you wish was easier?" You will be surprised. Most employees aren't hiding anything, they just haven't been asked. You cannot govern what you cannot see, and you can't see what you haven't asked about.

Two. Sort the work by sensitivity. Drafting an internal announcement, brainstorming a tagline, summarizing a public article, all low risk. Categorizing real vendor invoices, running variance on actual numbers, processing payroll data, drafting investor communications, all different. A short matrix with "what data" on one axis and "what destination" on the other covers most of it.

Three. Stand up approved tools. ChatGPT Enterprise or ChatGPT Business with an admin-managed workspace for general work. Claude for Enterprise if you want a second vendor.

For the accounting-specific jobs, use tools that already live inside your security boundary.

Dext for document processing. Ramp accounting automation for expense categorization. A2X accounting for ecommerce revenue flowing into Xero.

The AI built into Xero itself. These are AI-powered integrations that run inside your perimeter, not through someone's personal Gmail.

Four. Build approved AI into the workflow instead of bolting it on. This is where workflow optimization and operational efficiency actually come from. An AI-assisted review step baked into your n8n monthly close. AI extraction running inside your Dext AP flow. Meeting automation that captures notes without anyone uploading a Zoom recording to a random tool. Slack automation and email automation that handle the small stuff so nobody's pasting content into ChatGPT to draft a routine reply. When the governed path is the easier path, that's the path your team takes.

Five. Write a one-page policy. Not twenty pages. One page. Here are the tools we've approved. Here are the data categories that never go into consumer AI. Here's how to flag a new tool you want to try. Here's who to ask. A policy people can remember is a policy people follow.

Six. Revisit quarterly. AI is moving too fast for an annual review. New models, new features, new tools your team just discovered on TikTok. A thirty-minute review every quarter keeps the governance current without adding bureaucracy.

That's it. Six moves. None of them are heavy.

What a Good Answer Looks Like for a Founder

A few months ago I was on a call with a Series A founder whose head of ops had just left. She was doing the handoff with her interim controller, and the controller said, casually, "okay and we should probably talk about the ChatGPT thing."

Turned out the previous ops lead had been using ChatGPT for just about everything. Vendor contract summaries. Expense categorization. Investor update drafts. Close checklists. Runway models. All from a personal account. None of it was in a tracker anywhere.

The founder called me. We spent two hours doing what I just described above. Inventory, sorting, tool choices, workflow changes, policy. We moved three tasks that had been in ChatGPT into Dext, Xero's native automation, and a governed Claude for Enterprise workspace. We shut down the rest.

Total cost, including licenses: under $800 a month. Total elapsed time from "we have a problem" to "we have a system": eleven days.

That's what this looks like when someone takes it seriously. Not a crisis. A project.

The companies where this becomes a crisis are the ones where it comes up during SOC 2 scoping, or during investor diligence, or after a journalist emails asking about an AI-related incident. At that point, the conversation costs a lot more than $800 a month.

Frequently Asked Questions

Is it ever safe for employees to use ChatGPT for accounting work?

Yes, with the right setup. ChatGPT Enterprise or Business, clear rules about what data categories are in and out of bounds, and a human review step before anything AI-touched hits the books. The unsafe version is personal accounts on personal devices with no guardrails, and that's what most startups have today.

What's the difference between ChatGPT and ChatGPT Enterprise for compliance?

Enterprise is in OpenAI's SOC 2 Type II scope, doesn't train on your inputs by default, supports configurable retention, and offers Enterprise Key Management. Consumer ChatGPT doesn't. For a startup carrying SOC 2 obligations or customer data handling commitments, that distinction is not a detail.

How do I find out what AI tools my team is actually using?

Ask. Most people aren't hiding it. A short survey, a Slack thread, a fifteen-minute conversation with each function lead. If you want to go further, scan browser extensions, check installed apps, and look at expense reports for AI tool subscriptions. You'll see 80% of what's happening in under a day.

Can I just block ChatGPT on the company network?

You can. It usually pushes usage onto personal phones and home WiFi, which makes visibility worse. A blend of approved tools, clear policy, and governed AI built into workflows beats a hard block for almost every scaling startup.

What about AI hallucinations in financial reporting?

Treat any AI-generated number, accounting treatment, or tax position as a draft that needs human review against source data. Not sometimes. Every time. The tool writes with the same confidence whether it's right or wrong, and that confidence is what fools tired reviewers.

Does MATAX help with AI governance as part of its Xero accounting services?

Yes. AI governance sits right at the intersection of operational efficiency and compliance, which is what we do. We help clients see what's actually happening, stand up approved tools, build governed AI into their close and AP workflows, and line the whole stack up against their SOC 2 and investor diligence expectations.

The Line I Want You to Remember

A successful business in 2026 is going to use AI heavily. You don't get to opt out of that.

The question is whether the AI in your company runs through systems you designed, or through the personal ChatGPT accounts of employees trying to move faster than your infrastructure lets them.

The first path compounds. The work gets faster, the books get cleaner, the audit trail stays intact, and your team productivity goes up because the governance is invisible. The second path accumulates quiet risk until it isn't quiet anymore, and by then you're explaining it to someone.

We've worked with hundreds of startups on exactly this transition. The pattern is always the same. The companies that treat AI as part of their operations design get the productivity. The companies that treat it as a future compliance problem eventually get both the risk and the cleanup bill.

Start with the inventory. Everything else follows.

Dawn Hatch is the Founding Partner of MATAX, a San Francisco-based firm specializing in Xero implementation, AI workflow automation, and back office operations.