Heuristic Assertion Agent: Automating Schema-Free API Testing

Let's start with the basics: what's a heuristic?
Simply put, heuristics are intelligent rules-of-thumb, smart guesses or educated assumptions, used to quickly assess if something appears correct. Unlike strict schema-based assertions, heuristics allow intuitive validation of responses based on patterns, common sense, and prior experience.
For example, if an API returns a response with a field named is_active, a reasonable heuristic might be, "this field should probably be boolean". Or, if you see a field called email, a useful heuristic could be "this should be a non-empty string containing an '@' sign."
These are practical checks grounded in intuition, not strict rules.

Schema-based testing works well with comprehensive OpenAPI specifications, but reality isn't always so neat:

• Undocumented endpoints: Integrating a third-party API with limited documentation.
• Rapidly changing APIs: Backend teams iterating quickly without detailed schemas.
• Legacy systems: Old endpoints with outdated or incomplete documentation.

Heuristics excel in these scenarios, providing flexible coverage and confidence, even as APIs evolve.

The Heuristic Assertion Agent follows a straightforward idea:

Given a raw curl command, the agent will:

1. Execute the command and fetch the response.
2. Use an LLM (like GPT-4 or GPT-3.5 Turbo) to infer heuristic assertions based solely on the response.
3. Verify these assertions against the actual response data, using the LLM not only for verification but also to provide clear reasoning.
4. Generate easy-to-understand reports detailing pass/fail status along with explanations.

This approach involves no hardcoded rules or schemas. Instead, the LLM acts as an intelligent partner, intuitively generating and validating tests and clearly explaining its reasoning.

Our implementation uses the powerful orchestration tool LangGraph. Here's the workflow:

curl command → InputParser → Heuristic Generator → Assertion Verifier → Report Logger

• InputParser: Executes the curl command and captures the JSON response.
• Heuristic Generator: LLM generates intuitive assertions like:
• "Ensure the success field is boolean"
• "Check if the user_id is a non-empty integer"
• Assertion Verifier: Passes each heuristic back to the LLM, which not only evaluates pass/fail but provides detailed reasons explaining the evaluation.
• Report Logger: Clearly captures and logs these detailed evaluation results.

The result: a fully automated, intelligent, and schema-independent testing approach.

Several aspects stand out:

• Flexibility: Automatically adapts to evolving API structures.
• Intelligence: Uses LLM reasoning beyond simple checks, clearly explaining results.
• Speed: Immediate generation of comprehensive test coverage.
• Low Maintenance: Less brittle; heuristics evolve naturally with APIs.

Consider a simple API endpoint:

curl https://fakerestapi.azurewebsites.net/api/v1/Activities

The agent quickly:

• Fetches and analyzes the response JSON.
• Creates heuristic assertions:
• "The response array should contain objects with id, title, dueDate, and completed."
• "Each id should be a unique positive integer."
• Automatically verifies each assertion with detailed reasoning:
• ✅ All activity objects contain the keys 'id', 'title', 'dueDate', and 'completed'.
• ❌ Duplicate id values were found in the response array; IDs should be unique positive integers.

Instantly generated, detailed, and actionable results.

The heuristic assertion method isn't just a curiosity, it's a step toward smarter test automation. Future opportunities include:

• CI/CD Integration: Validate APIs continuously without manual intervention.
• Confidence Scoring: Prioritize heuristics based on reliability and model confidence.
• Automatic Bug Logging: Seamlessly integrate with defect tracking tools.
• Real-time Monitoring: Proactively detect and alert on API changes.

Heuristics combined with LLMs offer a powerful approach to API testing. They enhance traditional test automation by adding intelligent reasoning, flexibility, and resilience.
It's not about replacing human testers, it's about empowering them with tools that are dynamic, insightful, and effective.