ComfyUI/custom_nodes/ComfyUI-Impact-Pack/docs/wildcards/WILDCARD_TESTING_GUIDE.md

Wildcard System Testing Guide

Complete testing guide for the ComfyUI Impact Pack wildcard system.

---

📋 Table of Contents

1. Test Overview
2. Test Suites
3. Quick Start
4. Running Tests
5. Test Validation

---

Test Overview

Test Statistics

• Total Tests: 86 tests across 7 suites
• Coverage: 100% of PRD core requirements
• Pass Rate: 100%
• Test Types: UTF-8, error handling, edge cases, nesting, on-demand, config, dynamic prompts

Test Structure

tests/
├── Test Suites (7 suites, 86 tests)
│   ├── test_encoding.sh              # 15 tests - UTF-8 multi-language support
│   ├── test_error_handling.sh        # 10 tests - Error recovery and graceful handling
│   ├── test_edge_cases.sh            # 20 tests - Boundary conditions and special cases
│   ├── test_deep_nesting.sh          # 17 tests - 7-level transitive expansion + pattern matching
│   ├── test_ondemand_loading.sh      #  8 tests - Progressive lazy loading with cache limits
│   ├── test_config_quotes.sh         #  5 tests - Configuration path handling
│   └── test_dynamic_prompts_full.sh  # 11 tests - Weighted/multiselect with statistical validation
│
├── Documentation
│   ├── README.md                     # Test suite overview
│   └── RUN_ALL_TESTS.md              # Execution guide
│
├── Test Samples
│   └── wildcards/samples/            # Test wildcard files
│       ├── level1/.../level7/        # 7-level nesting structure
│       ├── *.txt                     # Various test wildcards
│       └── 아름다운색.txt             # Korean UTF-8 sample
│
└── Utilities
    └── restart_test_server.sh        # Server management utility

---

Test Suites

1. UTF-8 Encoding Tests (15 tests)

File: test_encoding.sh Port: 8188 Purpose: Multi-language support validation

Test Coverage:

• Korean text (한글)
• Chinese text (中文)
• Arabic text (العربية)
• Emoji support (🐉🔥⚡)
• Special characters
• Mixed multi-language content
• Case-insensitive Korean matching

Key Validations:

• All non-ASCII characters preserved
• UTF-8 encoding consistency
• No character corruption
• Proper string comparison

---

2. Error Handling Tests (10 tests)

File: test_error_handling.sh Port: 8189 Purpose: Graceful error recovery

Test Coverage:

• Non-existent wildcards
• Missing files
• Circular reference detection (direct and indirect)
• Malformed dynamic prompt syntax
• Deep nesting without crashes
• Invalid quantifiers

Key Validations:

• No server crashes
• Clear error messages
• Original text preserved on error
• Circular detection within 100 iterations

---

3. Edge Cases Tests (20 tests)

File: test_edge_cases.sh Port: 8190 Purpose: Boundary conditions and special scenarios

Test Coverage:

• Empty lines and comments in wildcard files
• Very long lines (>1000 chars)
• Basic wildcard expansion
• Case-insensitive matching
• Quantifiers (1-10 repetitions)
• Pattern matching (__*/name__)

Key Validations:

• Empty lines filtered correctly
• Comments ignored properly
• Long text handling
• Quantifier accuracy
• Pattern matching at any depth

---

4. Deep Nesting Tests (17 tests)

File: test_deep_nesting.sh Port: 8194 Purpose: 7-level transitive expansion and pattern matching

Test Coverage:

• Direct level access (Level 1-7)
• Transitive expansion through all levels
• Multiple wildcard nesting
• Mixed depth combinations
• Quantifiers with nesting
• Weighted selection with nesting
• Depth-agnostic pattern matching

Key Validations:

• All 7 levels fully expanded
• No unexpanded wildcards remain
• Pattern matching ignores directory depth
• Complex combinations work correctly

Directory Structure:

samples/level1/level2/level3/level4/level5/level6/level7/

---

5. On-Demand Loading Tests (8 tests)

File: test_ondemand_loading.sh Port: 8191 Purpose: Progressive lazy loading with configurable cache limits

Test Coverage:

• Small cache (1MB) - On-demand mode
• Medium cache (10MB) - Hybrid mode
• Large cache (100MB) - Full cache mode
• Aggressive lazy (0.5MB)
• Various thresholds (5MB, 20MB, 50MB)

Key Validations:

• Correct loading mode selection
• Progressive loading functionality
• Cache limit enforcement
• No performance degradation

Note: Uses temporary samples in /tmp/ with auto-cleanup

---

6. Config Quotes Tests (5 tests)

File: test_config_quotes.sh Port: 8192 Purpose: Configuration path handling with quotes

Test Coverage:

• Paths with single quotes
• Paths with double quotes
• Paths with spaces (quoted)
• Mixed quote scenarios
• Unquoted baseline

Key Validations:

• Quotes stripped correctly
• Paths with spaces handled
• Wildcards loaded from quoted paths

---

7. Dynamic Prompts Tests (11 tests)

File: test_dynamic_prompts_full.sh Port: 8193 Purpose: Statistical validation of weighted and multiselect features

Test Coverage:

• Multiselect (2-5 items) with custom separators
• Weighted selection (various ratios: 10:1, 1:1:1, 5:3:2)
• Nested dynamic prompts
• Basic random selection
• Seed variation validation

Statistical Validation:

• 100 iterations for weighted selection
• 20 iterations for multiselect
• Distribution verification (±15% tolerance)
• Duplicate detection
• Separator validation

Key Validations:

• Exact item count for multiselect
• No duplicates in multiselect
• Correct separators
• Statistical distribution matches weight ratios
• Nested prompt expansion

---

Quick Start

Run All Tests

cd tests/
bash test_encoding.sh && \
bash test_error_handling.sh && \
bash test_edge_cases.sh && \
bash test_deep_nesting.sh && \
bash test_ondemand_loading.sh && \
bash test_config_quotes.sh && \
bash test_dynamic_prompts_full.sh

Run Individual Suite

cd tests/
bash test_encoding.sh

Check Test Results

All tests output:

• ✅ PASS - Test succeeded with validation
• ❌ FAIL - Test failed (should not occur)
• ⚠️ WARNING - Partial success or non-critical issue

---

Running Tests

Prerequisites

• ComfyUI server must be installable
• Port availability (8188-8194)
• Network access to 127.0.0.1
• Python 3 with json module

Automatic Server Management

All test suites automatically:

1. Kill any existing server on target port
2. Create temporary configuration file
3. Start ComfyUI server
4. Wait for server ready (up to 60s)
5. Execute tests
6. Clean up (kill server, remove config)

Test Execution Flow

1. Setup
   ├─ Kill existing server on port
   ├─ Create impact-pack.ini config
   └─ Start ComfyUI server

2. Wait for Ready
   ├─ Poll server every second
   ├─ Max 60 seconds timeout
   └─ Log tail on failure

3. Execute Tests
   ├─ Call /impact/wildcards API
   ├─ Validate responses
   └─ Check behavior

4. Cleanup
   ├─ Kill server process
   └─ Remove config file

---

Test Validation

What Tests Validate

Behavioral Validation (Not just "no errors"):

• Weighted Selection: Statistical distribution matches weight ratios
• Multiselect: Exact count, no duplicates, correct separator
• Nesting: All levels fully expanded, no remaining wildcards
• Pattern Matching: Depth-agnostic matching works correctly
• UTF-8: Character preservation and proper encoding
• Error Handling: Graceful recovery with meaningful messages

Success Criteria

• All 86 tests must pass (100% pass rate)
• No server crashes or hangs
• API responses within expected format
• Statistical distributions within ±15% tolerance
• No unexpanded wildcards in final output

Validation Examples

Weighted Selection:

# Test 10:1 ratio with 100 iterations
# Expected: ~91% common, ~9% rare
# Actual: Count distribution within ±15%

Multiselect:

# Test {2$$, $$red|blue|green}
# Expected: Exactly 2 items, comma-space separator, no duplicates
# Validation: Count words, check separator, detect duplicates

Pattern Matching:

# Test __*/dragon__
# Expected: Matches dragon.txt, fantasy/dragon.txt, dragon/fire.txt
# Validation: No unexpanded wildcards remain

---

Troubleshooting

Common Issues

Server Fails to Start:

# Check log file
tail -20 /tmp/{test_name}_test.log

# Check port availability
lsof -i :8188

# Kill conflicting process
pkill -f "python.*main.py.*--port 8188"

Tests Timeout:

• Increase wait time in test script (default 60s)
• Check server performance and resources
• Verify network connectivity to 127.0.0.1

Statistical Tests Fail:

• Expected for very small sample sizes
• ±15% tolerance accounts for randomness
• Rerun test to verify consistency

UTF-8 Issues:

• Ensure terminal supports UTF-8
• Check file encoding: file -i tests/wildcards/samples/*.txt
• Verify locale: locale | grep UTF-8

---

Test Maintenance

Adding New Tests

1. Create new test function in appropriate suite
2. Follow existing test patterns (setup, execute, validate, cleanup)
3. Update test counts in README.md and SUMMARY.md
4. Update this guide with new test description

Modifying Existing Tests

1. Preserve behavioral validation (not just "no errors")
2. Maintain statistical rigor for dynamic prompt tests
3. Update documentation if test purpose changes
4. Verify all 86 tests still pass after modification

Test Philosophy

• Tests validate behavior, not just execution success
• Statistical validation for probabilistic features
• Real-world scenarios with production-like setup
• Comprehensive coverage of all PRD requirements