# Testing Strategies with AI: A Practical Guide to Quality Assurance in Vibe Coding
Testing with AI assistance isn't about having AI write all your tests and calling it done. It's about leveraging AI to write better tests faster, catch edge cases you might miss, and maintain comprehensive test coverage as your codebase evolves. In this lesson, you'll learn how to strategically use AI to build robust testing strategies that actually work.
## The AI Testing Paradox
Here's the challenge: AI can write tests quickly, but it can also write tests that pass without actually testing anything meaningful. I've reviewed countless PRs where developers proudly show 90% coverage, but the tests only verify that functions return *something*, not that they return the *right* thing.
The key is using AI as a collaborative partner in your testing strategy, not a replacement for critical thinking about what needs testing and why.
## Starting with Test Strategy, Not Test Code
Before asking AI to write a single test, establish your testing strategy. This is where AI excels as a brainstorming partner.
### Identifying What to Test
Start by having a conversation with your AI assistant about what aspects of your code need testing:
```markdown
I have a payment processing module that handles:
- Credit card validation
- Payment gateway integration
- Retry logic for failed payments
- Refund processing
- Currency conversion
What testing strategy would you recommend? What are the critical paths
and edge cases I should focus on?
```
AI assistants are excellent at identifying edge cases you might overlook. They'll suggest scenarios like:
- Network timeouts during payment processing
- Partial refunds across currency boundaries
- Race conditions in retry logic
- Invalid card numbers that pass basic validation
This strategic conversation is more valuable than immediately jumping to "write me tests for this function." For more on maintaining quality standards, see our lesson on [quality-control](/lessons/quality-control).
## Writing Unit Tests with AI
Once you have your strategy, AI can dramatically accelerate unit test creation. But you need to be specific about what constitutes a good test.
### The Specific Prompt Pattern
Compare these two prompts:
**Vague prompt:**
```markdown
Write tests for this function.
```
**Effective prompt:**
```markdown
Write unit tests for this validatePaymentCard function. Include:
1. Tests for valid Visa, Mastercard, and Amex cards
2. Tests for invalid card numbers (wrong length, invalid Luhn checksum)
3. Tests for expired cards
4. Tests for cards with invalid CVV
5. Mock the external validation API
6. Use Jest and ensure each test case is isolated
```
The second prompt produces tests that actually verify behavior:
```javascript
import { validatePaymentCard } from './payment';
import { cardValidationAPI } from './external-apis';
jest.mock('./external-apis');
describe('validatePaymentCard', () => {
beforeEach(() => {
jest.clearAllMocks();
});
describe('valid cards', () => {
it('should accept valid Visa card', async () => {
const card = {
number: '4532015112830366',
expiry: '12/25',
cvv: '123'
};
cardValidationAPI.verify.mockResolvedValue({ valid: true });
const result = await validatePaymentCard(card);
expect(result.isValid).toBe(true);
expect(cardValidationAPI.verify).toHaveBeenCalledWith(
expect.objectContaining({ number: card.number })
);
});
it('should accept valid Mastercard', async () => {
const card = {
number: '5425233430109903',
expiry: '06/24',
cvv: '456'
};
cardValidationAPI.verify.mockResolvedValue({ valid: true });
const result = await validatePaymentCard(card);
expect(result.isValid).toBe(true);
expect(result.cardType).toBe('mastercard');
});
});
describe('invalid cards', () => {
it('should reject card with invalid Luhn checksum', async () => {
const card = {
number: '4532015112830367', // Last digit wrong
expiry: '12/25',
cvv: '123'
};
const result = await validatePaymentCard(card);
expect(result.isValid).toBe(false);
expect(result.error).toBe('Invalid card number');
expect(cardValidationAPI.verify).not.toHaveBeenCalled();
});
it('should reject expired card', async () => {
const card = {
number: '4532015112830366',
expiry: '12/20', // Expired
cvv: '123'
};
const result = await validatePaymentCard(card);
expect(result.isValid).toBe(false);
expect(result.error).toBe('Card expired');
});
it('should reject card with invalid CVV length', async () => {
const card = {
number: '4532015112830366',
expiry: '12/25',
cvv: '12' // Too short
};
const result = await validatePaymentCard(card);
expect(result.isValid).toBe(false);
expect(result.error).toContain('CVV');
});
});
describe('API interaction', () => {
it('should handle API timeout gracefully', async () => {
const card = {
number: '4532015112830366',
expiry: '12/25',
cvv: '123'
};
cardValidationAPI.verify.mockRejectedValue(
new Error('Request timeout')
);
const result = await validatePaymentCard(card);
expect(result.isValid).toBe(false);
expect(result.error).toBe('Validation service unavailable');
});
});
});
```
## Integration Testing with AI
Integration tests verify that components work together correctly. AI is particularly helpful here because it can help you identify integration points and potential failure modes.
### Mapping Integration Points
Ask AI to analyze your system:
```markdown
Here's my e-commerce checkout flow:
1. Cart validation
2. Inventory check
3. Payment processing
4. Order creation
5. Inventory update
6. Email confirmation
What integration tests should I write? What failure scenarios
should I test?
```
AI will suggest scenarios like:
- Payment succeeds but inventory update fails
- Email service is down
- Race condition: item sold out between check and purchase
Then have AI help scaffold the tests:
```javascript
import { processCheckout } from './checkout';
import { inventoryService } from './inventory';
import { paymentService } from './payment';
import { emailService } from './email';
describe('Checkout Integration Tests', () => {
it('should complete full checkout flow successfully', async () => {
const cart = [
{ productId: 'PROD-1', quantity: 2 },
{ productId: 'PROD-2', quantity: 1 }
];
// This is a real integration test - no mocking
const result = await processCheckout({
cart,
payment: { /* payment details */ },
customer: { email: 'test@example.com' }
});
expect(result.orderId).toBeDefined();
expect(result.status).toBe('completed');
// Verify side effects
const inventory = await inventoryService.getStock('PROD-1');
expect(inventory.reserved).toBeGreaterThan(0);
// Verify email was queued
const emailQueue = await emailService.getQueue();
expect(emailQueue).toContainEqual(
expect.objectContaining({
to: 'test@example.com',
type: 'order_confirmation'
})
);
});
it('should rollback on payment failure', async () => {
const cart = [{ productId: 'PROD-1', quantity: 2 }];
const initialStock = await inventoryService.getStock('PROD-1');
// Inject a payment failure
paymentService.setTestMode('decline');
await expect(
processCheckout({
cart,
payment: { /* payment details */ },
customer: { email: 'test@example.com' }
})
).rejects.toThrow('Payment declined');
// Verify inventory was not affected
const finalStock = await inventoryService.getStock('PROD-1');
expect(finalStock.available).toBe(initialStock.available);
// Verify no order was created
const orders = await orderService.getByCustomer('test@example.com');
expect(orders).toHaveLength(0);
});
});
```
## Test-Driven Development with AI
AI can supercharge TDD by helping you write tests first, then implement to pass them. This is covered more in [testing-strategies](/lessons/testing-strategies), but here's a practical workflow:
### The AI-Assisted TDD Loop
1. **Describe the feature to AI:**
```markdown
I need to implement a rate limiter that:
- Allows 100 requests per minute per user
- Returns time until next allowed request when limit exceeded
- Resets counter every minute
- Works across multiple servers (use Redis)
Write failing tests for this first.
```
2. **Review and refine the generated tests**
3. **Ask AI to implement the code to pass the tests**
4. **Review, refactor, and verify**
This approach ensures you think through the API design before implementation.
## Testing AI-Generated Code
When AI generates code, you need extra vigilance in testing. AI can produce code that works for the happy path but fails on edge cases. Check our [hallucination-detection](/lessons/hallucination-detection) lesson for more on this.
### The Verification Checklist
After AI generates code, use AI to generate adversarial tests:
```markdown
Here's the code you just generated. Now write tests that try to BREAK it.
What edge cases, invalid inputs, or race conditions could cause problems?
```
This produces tests like:
```javascript
describe('adversarial tests for AI-generated parser', () => {
it('should handle deeply nested structures without stack overflow', () => {
const deeplyNested = { a: { b: { c: { /* 1000 levels deep */ } } } };
expect(() => parser.parse(deeplyNested)).not.toThrow();
});
it('should handle circular references', () => {
const obj = { name: 'test' };
obj.self = obj;
expect(() => parser.parse(obj)).not.toThrow();
});
it('should handle extremely large numbers', () => {
const data = { value: Number.MAX_SAFE_INTEGER + 1 };
const result = parser.parse(data);
expect(result.value).toBe(data.value);
});
});
```
## Property-Based Testing with AI
AI is excellent at helping you implement property-based tests, which verify that your code maintains certain properties regardless of input.
```javascript
import fc from 'fast-check';
import { sortArray } from './sorting';
describe('sortArray properties', () => {
it('should maintain array length', () => {
fc.assert(
fc.property(fc.array(fc.integer()), (arr) => {
const sorted = sortArray([...arr]);
return sorted.length === arr.length;
})
);
});
it('should contain same elements as input', () => {
fc.assert(
fc.property(fc.array(fc.integer()), (arr) => {
const sorted = sortArray([...arr]);
const sortedCopy = [...sorted].sort((a, b) => a - b);
const originalSorted = [...arr].sort((a, b) => a - b);
return JSON.stringify(sortedCopy) === JSON.stringify(originalSorted);
})
);
});
it('should produce sorted output', () => {
fc.assert(
fc.property(fc.array(fc.integer()), (arr) => {
const sorted = sortArray([...arr]);
for (let i = 1; i < sorted.length; i++) {
if (sorted[i] < sorted[i - 1]) return false;
}
return true;
})
);
});
});
```
Ask AI: "Generate property-based tests for [your function] that verify invariants like idempotency, reversibility, or ordering preservation."
## Maintaining Test Suites Over Time
As your codebase evolves, tests need maintenance. AI can help identify outdated tests and suggest updates. This ties into [managing-tech-debt](/lessons/managing-tech-debt).
### The Test Audit Prompt
```markdown
I've refactored my authentication module. Here are the old tests and
the new code. Which tests are now invalid? Which new scenarios need testing?
What tests need updating?
OLD TESTS:
[paste tests]
NEW CODE:
[paste code]
```
AI will identify:
- Tests that no longer match the API
- Missing tests for new functionality
- Tests that are now redundant
- Assertions that need updating
## Performance Testing with AI
AI can help you write performance tests and identify bottlenecks:
```javascript
import { performance } from 'perf_hooks';
import { processLargeDataset } from './data-processor';
describe('performance tests', () => {
it('should process 10k records in under 1 second', async () => {
const records = generateTestRecords(10000);
const start = performance.now();
await processLargeDataset(records);
const end = performance.now();
const duration = end - start;
expect(duration).toBeLessThan(1000);
});
it('should handle concurrent processing efficiently', async () => {
const batches = Array(10).fill(null).map(() =>
generateTestRecords(1000)
);
const start = performance.now();
await Promise.all(batches.map(batch => processLargeDataset(batch)));
const end = performance.now();
// Should be faster than sequential (10s) due to concurrency
expect(end - start).toBeLessThan(3000);
});
});
```
For more on performance considerations, see [performance-optimization](/lessons/performance-optimization).
## Common AI Testing Pitfalls
Based on reviewing hundreds of AI-assisted test suites, here are the most common mistakes:
### 1. Over-Mocking
AI tends to mock everything. Ask it explicitly: "Which of these dependencies should be real and which should be mocked? Why?"
### 2. Testing Implementation Details
AI often writes tests that verify how something works rather than what it produces:
```javascript
// Bad: Tests implementation
it('should call internal _processData method', () => {
const spy = jest.spyOn(service, '_processData');
service.execute();
expect(spy).toHaveBeenCalled();
});
// Good: Tests behavior
it('should return processed data', () => {
const result = service.execute();
expect(result).toMatchObject({ processed: true, data: expect.any(Array) });
});
```
### 3. Insufficient Assertions
AI might write tests that verify something happened but not that the right thing happened:
```javascript
// Bad: Vague assertion
it('should process payment', async () => {
const result = await processPayment(payment);
expect(result).toBeDefined();
});
// Good: Specific assertions
it('should process payment and return transaction details', async () => {
const result = await processPayment(payment);
expect(result).toMatchObject({
success: true,
transactionId: expect.stringMatching(/^txn_/),
amount: payment.amount,
currency: payment.currency,
timestamp: expect.any(Number)
});
});
```
For more common mistakes, check [top-mistakes](/lessons/top-mistakes).
## Team Testing Workflows
When working with a team, establish conventions for AI-assisted testing:
1. **Standardize prompts**: Create a team library of testing prompts
2. **Code review AI tests**: Don't merge without human review
3. **Coverage thresholds**: Set minimums but don't worship 100%
4. **Test naming conventions**: Ensure AI follows your team's patterns
See [team-workflows](/lessons/team-workflows) for more collaboration strategies.
## When Not to Use AI for Testing
Some scenarios are better handled without AI assistance:
- **Security-critical tests**: Write these yourself and have security experts review
- **Complex business logic**: You understand the domain better than AI
- **Visual regression tests**: AI can't see your UI
- **Exploratory testing**: Human creativity is essential
Our [when-not-to-use-ai](/lessons/when-not-to-use-ai) lesson covers this in depth.
## Practical Exercise: Build a Test Suite
Let's put this into practice. Take a module from your codebase and:
1. Ask AI to analyze it and suggest a testing strategy
2. Write 3 unit tests with AI assistance
3. Write 1 integration test that verifies component interaction
4. Ask AI to generate adversarial tests
5. Review all tests and refine the assertions
6. Run the suite and fix any failures
This exercise should take 30-45 minutes and will give you practical experience with the patterns covered in this lesson.
## Key Takeaways
- **Strategy first, code second**: Use AI to brainstorm testing approaches before writing tests
- **Be specific in prompts**: Vague prompts produce useless tests
- **Verify AI-generated tests**: Review assertions, check edge cases, ensure they actually test something meaningful
- **Use AI for test maintenance**: Let AI help identify outdated tests and suggest updates
- **Combine AI strengths with human judgment**: AI finds edge cases; you understand business logic
- **Don't trust coverage metrics blindly**: 90% coverage with poor assertions is worse than 60% with good ones
Testing with AI is about augmenting your testing practice, not replacing it. The best test suites come from developers who use AI to write more tests, better tests, and to catch scenarios they might have missed—while still applying critical thinking to ensure those tests actually verify the right things.
In the next lesson on [doc-generation](/lessons/doc-generation), we'll explore how to use AI to create comprehensive documentation that makes your well-tested code easy to understand and maintain.