Testing Strategies with AI: A Practical Guide to Quality Assurance in Vibe Coding

Testing with AI assistance isn't about having AI write all your tests and calling it done. It's about leveraging AI to write better tests faster, catch edge cases you might miss, and maintain comprehensive test coverage as your codebase evolves. In this lesson, you'll learn how to strategically use AI to build robust testing strategies that actually work.

The AI Testing Paradox

Here's the challenge: AI can write tests quickly, but it can also write tests that pass without actually testing anything meaningful. I've reviewed countless PRs where developers proudly show 90% coverage, but the tests only verify that functions return something, not that they return the right thing.

The key is using AI as a collaborative partner in your testing strategy, not a replacement for critical thinking about what needs testing and why.

Starting with Test Strategy, Not Test Code

Before asking AI to write a single test, establish your testing strategy. This is where AI excels as a brainstorming partner.

Identifying What to Test

Start by having a conversation with your AI assistant about what aspects of your code need testing:

I have a payment processing module that handles:
- Credit card validation
- Payment gateway integration
- Retry logic for failed payments
- Refund processing
- Currency conversion

What testing strategy would you recommend? What are the critical paths
and edge cases I should focus on?

AI assistants are excellent at identifying edge cases you might overlook. They'll suggest scenarios like:

Network timeouts during payment processing
Partial refunds across currency boundaries
Race conditions in retry logic
Invalid card numbers that pass basic validation

This strategic conversation is more valuable than immediately jumping to "write me tests for this function." For more on maintaining quality standards, see our lesson on quality-control.

Writing Unit Tests with AI

Once you have your strategy, AI can dramatically accelerate unit test creation. But you need to be specific about what constitutes a good test.

The Specific Prompt Pattern

Compare these two prompts:

Vague prompt:

Write tests for this function.

Effective prompt:

Write unit tests for this validatePaymentCard function. Include:
1. Tests for valid Visa, Mastercard, and Amex cards
2. Tests for invalid card numbers (wrong length, invalid Luhn checksum)
3. Tests for expired cards
4. Tests for cards with invalid CVV
5. Mock the external validation API
6. Use Jest and ensure each test case is isolated

The second prompt produces tests that actually verify behavior:

import { validatePaymentCard } from './payment';
import { cardValidationAPI } from './external-apis';

jest.mock('./external-apis');

describe('validatePaymentCard', () => {
  beforeEach(() => {
    jest.clearAllMocks();
  });

  describe('valid cards', () => {
    it('should accept valid Visa card', async () => {
      const card = {
        number: '4532015112830366',
        expiry: '12/25',
        cvv: '123'
      };
      
      cardValidationAPI.verify.mockResolvedValue({ valid: true });
      
      const result = await validatePaymentCard(card);
      
      expect(result.isValid).toBe(true);
      expect(cardValidationAPI.verify).toHaveBeenCalledWith(
        expect.objectContaining({ number: card.number })
      );
    });

    it('should accept valid Mastercard', async () => {
      const card = {
        number: '5425233430109903',
        expiry: '06/24',
        cvv: '456'
      };
      
      cardValidationAPI.verify.mockResolvedValue({ valid: true });
      
      const result = await validatePaymentCard(card);
      
      expect(result.isValid).toBe(true);
      expect(result.cardType).toBe('mastercard');
    });
  });

  describe('invalid cards', () => {
    it('should reject card with invalid Luhn checksum', async () => {
      const card = {
        number: '4532015112830367', // Last digit wrong
        expiry: '12/25',
        cvv: '123'
      };
      
      const result = await validatePaymentCard(card);
      
      expect(result.isValid).toBe(false);
      expect(result.error).toBe('Invalid card number');
      expect(cardValidationAPI.verify).not.toHaveBeenCalled();
    });

    it('should reject expired card', async () => {
      const card = {
        number: '4532015112830366',
        expiry: '12/20', // Expired
        cvv: '123'
      };
      
      const result = await validatePaymentCard(card);
      
      expect(result.isValid).toBe(false);
      expect(result.error).toBe('Card expired');
    });

    it('should reject card with invalid CVV length', async () => {
      const card = {
        number: '4532015112830366',
        expiry: '12/25',
        cvv: '12' // Too short
      };
      
      const result = await validatePaymentCard(card);
      
      expect(result.isValid).toBe(false);
      expect(result.error).toContain('CVV');
    });
  });

  describe('API interaction', () => {
    it('should handle API timeout gracefully', async () => {
      const card = {
        number: '4532015112830366',
        expiry: '12/25',
        cvv: '123'
      };
      
      cardValidationAPI.verify.mockRejectedValue(
        new Error('Request timeout')
      );
      
      const result = await validatePaymentCard(card);
      
      expect(result.isValid).toBe(false);
      expect(result.error).toBe('Validation service unavailable');
    });
  });
});

Integration Testing with AI

Integration tests verify that components work together correctly. AI is particularly helpful here because it can help you identify integration points and potential failure modes.

Mapping Integration Points

Ask AI to analyze your system:

Here's my e-commerce checkout flow:
1. Cart validation
2. Inventory check
3. Payment processing
4. Order creation
5. Inventory update
6. Email confirmation

What integration tests should I write? What failure scenarios 
should I test?

AI will suggest scenarios like:

Payment succeeds but inventory update fails
Email service is down
Race condition: item sold out between check and purchase

Then have AI help scaffold the tests:

import { processCheckout } from './checkout';
import { inventoryService } from './inventory';
import { paymentService } from './payment';
import { emailService } from './email';

describe('Checkout Integration Tests', () => {
  it('should complete full checkout flow successfully', async () => {
    const cart = [
      { productId: 'PROD-1', quantity: 2 },
      { productId: 'PROD-2', quantity: 1 }
    ];
    
    // This is a real integration test - no mocking
    const result = await processCheckout({
      cart,
      payment: { /* payment details */ },
      customer: { email: 'test@example.com' }
    });
    
    expect(result.orderId).toBeDefined();
    expect(result.status).toBe('completed');
    
    // Verify side effects
    const inventory = await inventoryService.getStock('PROD-1');
    expect(inventory.reserved).toBeGreaterThan(0);
    
    // Verify email was queued
    const emailQueue = await emailService.getQueue();
    expect(emailQueue).toContainEqual(
      expect.objectContaining({
        to: 'test@example.com',
        type: 'order_confirmation'
      })
    );
  });

  it('should rollback on payment failure', async () => {
    const cart = [{ productId: 'PROD-1', quantity: 2 }];
    
    const initialStock = await inventoryService.getStock('PROD-1');
    
    // Inject a payment failure
    paymentService.setTestMode('decline');
    
    await expect(
      processCheckout({
        cart,
        payment: { /* payment details */ },
        customer: { email: 'test@example.com' }
      })
    ).rejects.toThrow('Payment declined');
    
    // Verify inventory was not affected
    const finalStock = await inventoryService.getStock('PROD-1');
    expect(finalStock.available).toBe(initialStock.available);
    
    // Verify no order was created
    const orders = await orderService.getByCustomer('test@example.com');
    expect(orders).toHaveLength(0);
  });
});

Test-Driven Development with AI

AI can supercharge TDD by helping you write tests first, then implement to pass them. This is covered more in testing-strategies, but here's a practical workflow:

The AI-Assisted TDD Loop

Describe the feature to AI:

I need to implement a rate limiter that:
- Allows 100 requests per minute per user
- Returns time until next allowed request when limit exceeded
- Resets counter every minute
- Works across multiple servers (use Redis)

Write failing tests for this first.

Review and refine the generated tests
Ask AI to implement the code to pass the tests
Review, refactor, and verify

This approach ensures you think through the API design before implementation.

Testing AI-Generated Code

When AI generates code, you need extra vigilance in testing. AI can produce code that works for the happy path but fails on edge cases. Check our hallucination-detection lesson for more on this.

The Verification Checklist

After AI generates code, use AI to generate adversarial tests:

Here's the code you just generated. Now write tests that try to BREAK it.
What edge cases, invalid inputs, or race conditions could cause problems?

This produces tests like:

describe('adversarial tests for AI-generated parser', () => {
  it('should handle deeply nested structures without stack overflow', () => {
    const deeplyNested = { a: { b: { c: { /* 1000 levels deep */ } } } };
    expect(() => parser.parse(deeplyNested)).not.toThrow();
  });

  it('should handle circular references', () => {
    const obj = { name: 'test' };
    obj.self = obj;
    expect(() => parser.parse(obj)).not.toThrow();
  });

  it('should handle extremely large numbers', () => {
    const data = { value: Number.MAX_SAFE_INTEGER + 1 };
    const result = parser.parse(data);
    expect(result.value).toBe(data.value);
  });
});

Property-Based Testing with AI

AI is excellent at helping you implement property-based tests, which verify that your code maintains certain properties regardless of input.

import fc from 'fast-check';
import { sortArray } from './sorting';

describe('sortArray properties', () => {
  it('should maintain array length', () => {
    fc.assert(
      fc.property(fc.array(fc.integer()), (arr) => {
        const sorted = sortArray([...arr]);
        return sorted.length === arr.length;
      })
    );
  });

  it('should contain same elements as input', () => {
    fc.assert(
      fc.property(fc.array(fc.integer()), (arr) => {
        const sorted = sortArray([...arr]);
        const sortedCopy = [...sorted].sort((a, b) => a - b);
        const originalSorted = [...arr].sort((a, b) => a - b);
        return JSON.stringify(sortedCopy) === JSON.stringify(originalSorted);
      })
    );
  });

  it('should produce sorted output', () => {
    fc.assert(
      fc.property(fc.array(fc.integer()), (arr) => {
        const sorted = sortArray([...arr]);
        for (let i = 1; i < sorted.length; i++) {
          if (sorted[i] < sorted[i - 1]) return false;
        }
        return true;
      })
    );
  });
});

Ask AI: "Generate property-based tests for [your function] that verify invariants like idempotency, reversibility, or ordering preservation."

Maintaining Test Suites Over Time

As your codebase evolves, tests need maintenance. AI can help identify outdated tests and suggest updates. This ties into managing-tech-debt.

The Test Audit Prompt

I've refactored my authentication module. Here are the old tests and 
the new code. Which tests are now invalid? Which new scenarios need testing?
What tests need updating?

OLD TESTS:
[paste tests]

NEW CODE:
[paste code]

AI will identify:

Tests that no longer match the API
Missing tests for new functionality
Tests that are now redundant
Assertions that need updating

Performance Testing with AI

AI can help you write performance tests and identify bottlenecks:

import { performance } from 'perf_hooks';
import { processLargeDataset } from './data-processor';

describe('performance tests', () => {
  it('should process 10k records in under 1 second', async () => {
    const records = generateTestRecords(10000);
    
    const start = performance.now();
    await processLargeDataset(records);
    const end = performance.now();
    
    const duration = end - start;
    expect(duration).toBeLessThan(1000);
  });

  it('should handle concurrent processing efficiently', async () => {
    const batches = Array(10).fill(null).map(() => 
      generateTestRecords(1000)
    );
    
    const start = performance.now();
    await Promise.all(batches.map(batch => processLargeDataset(batch)));
    const end = performance.now();
    
    // Should be faster than sequential (10s) due to concurrency
    expect(end - start).toBeLessThan(3000);
  });
});

For more on performance considerations, see performance-optimization.

Common AI Testing Pitfalls

Based on reviewing hundreds of AI-assisted test suites, here are the most common mistakes:

1. Over-Mocking

AI tends to mock everything. Ask it explicitly: "Which of these dependencies should be real and which should be mocked? Why?"

2. Testing Implementation Details

AI often writes tests that verify how something works rather than what it produces:

// Bad: Tests implementation
it('should call internal _processData method', () => {
  const spy = jest.spyOn(service, '_processData');
  service.execute();
  expect(spy).toHaveBeenCalled();
});

// Good: Tests behavior
it('should return processed data', () => {
  const result = service.execute();
  expect(result).toMatchObject({ processed: true, data: expect.any(Array) });
});

3. Insufficient Assertions

AI might write tests that verify something happened but not that the right thing happened:

// Bad: Vague assertion
it('should process payment', async () => {
  const result = await processPayment(payment);
  expect(result).toBeDefined();
});

// Good: Specific assertions
it('should process payment and return transaction details', async () => {
  const result = await processPayment(payment);
  expect(result).toMatchObject({
    success: true,
    transactionId: expect.stringMatching(/^txn_/),
    amount: payment.amount,
    currency: payment.currency,
    timestamp: expect.any(Number)
  });
});

For more common mistakes, check top-mistakes.

Team Testing Workflows

When working with a team, establish conventions for AI-assisted testing:

Standardize prompts: Create a team library of testing prompts
Code review AI tests: Don't merge without human review
Coverage thresholds: Set minimums but don't worship 100%
Test naming conventions: Ensure AI follows your team's patterns

See team-workflows for more collaboration strategies.

When Not to Use AI for Testing

Some scenarios are better handled without AI assistance:

Security-critical tests: Write these yourself and have security experts review
Complex business logic: You understand the domain better than AI
Visual regression tests: AI can't see your UI
Exploratory testing: Human creativity is essential

Our when-not-to-use-ai lesson covers this in depth.

Practical Exercise: Build a Test Suite

Let's put this into practice. Take a module from your codebase and:

Ask AI to analyze it and suggest a testing strategy
Write 3 unit tests with AI assistance
Write 1 integration test that verifies component interaction
Ask AI to generate adversarial tests
Review all tests and refine the assertions
Run the suite and fix any failures

This exercise should take 30-45 minutes and will give you practical experience with the patterns covered in this lesson.

Key Takeaways

Strategy first, code second: Use AI to brainstorm testing approaches before writing tests
Be specific in prompts: Vague prompts produce useless tests
Verify AI-generated tests: Review assertions, check edge cases, ensure they actually test something meaningful
Use AI for test maintenance: Let AI help identify outdated tests and suggest updates
Combine AI strengths with human judgment: AI finds edge cases; you understand business logic
Don't trust coverage metrics blindly: 90% coverage with poor assertions is worse than 60% with good ones

Testing with AI is about augmenting your testing practice, not replacing it. The best test suites come from developers who use AI to write more tests, better tests, and to catch scenarios they might have missed—while still applying critical thinking to ensure those tests actually verify the right things.

In the next lesson on doc-generation, we'll explore how to use AI to create comprehensive documentation that makes your well-tested code easy to understand and maintain.