ComfyUI/custom_nodes/ComfyUI-Impact-Pack/tests/wildcards/README.md

# Wildcard System - Complete Test Suite

Comprehensive testing guide for the ComfyUI Impact Pack wildcard system.

---

## 📋 Quick Links

- **[Quick Start](#quick-start)** - Run tests in 5 minutes
- **[Test Categories](#test-categories)** - All test types
- **[Test Execution](#test-execution)** - How to run each test
- **[Troubleshooting](#troubleshooting)** - Common issues

---

## Overview

### Test Suite Structure

```
tests/
├── wildcards/                             # Wildcard system tests
│   ├── Unit Tests (Python)
│   │   ├── test_wildcard_lazy_loading.py      # LazyWildcardLoader class
│   │   ├── test_progressive_loading.py        # Progressive loading
│   │   ├── test_wildcard_final.py             # Final validation
│   │   └── test_lazy_load_verification.py     # Lazy load verification
│   │
│   ├── Integration Tests (Shell + API)
│   │   ├── test_progressive_ondemand.sh       # ⭐ Progressive loading (NEW)
│   │   ├── test_lazy_load_api.sh              # Lazy loading consistency
│   │   ├── test_sequential_loading.sh         # Transitive wildcards
│   │   ├── test_versatile_prompts.sh          # Feature tests
│   │   ├── test_wildcard_consistency.sh       # Consistency validation
│   │   └── test_wildcard_features.sh          # Core features
│   │
│   ├── Utility Scripts
│   │   ├── find_transitive_wildcards.sh       # Find transitive chains
│   │   ├── find_deep_transitive.py            # Deep transitive analysis
│   │   ├── verify_ondemand_mode.sh            # Verify on-demand activation
│   │   └── run_quick_test.sh                  # Quick validation
│   │
│   └── README.md (this file)
│
└── workflows/                             # Workflow test files
    ├── advanced-sampler.json
    ├── detailer-pipe-test.json
    └── ...
```

### Test Coverage

- **11 test files** (4 Python, 7 Shell)
- **100+ test scenarios**
- **~95% feature coverage**
- **~15 minutes** total execution time

---

## Quick Start

### Run All Tests

```bash
cd /path/to/ComfyUI/custom_nodes/comfyui-impact-pack/tests/wildcards

# Run all shell tests
for test in test_*.sh; do
    echo "Running: $test"
    bash "$test"
done
```

### Run Specific Test

```bash
cd /path/to/ComfyUI/custom_nodes/comfyui-impact-pack/tests/wildcards

# Progressive loading (NEW)
bash test_progressive_ondemand.sh

# Lazy loading
bash test_lazy_load_api.sh

# Sequential/transitive
bash test_sequential_loading.sh

# Versatile prompts
bash test_versatile_prompts.sh
```

---

## Test Categories

### 1. Progressive On-Demand Loading Tests ⭐ NEW

**Purpose**: Verify wildcards are loaded progressively as accessed.

**Test Files**:
- `test_progressive_ondemand.sh` (Shell, ~2 min)
- `test_progressive_loading.py` (Python unit test)

#### What's Tested

**Early Termination Size Calculation**:
```python
# Problem: 10GB scan takes 10-30 minutes
# Solution: Stop at cache limit
calculate_directory_size(path, limit=50MB)  # < 1 second
```

**YAML Pre-loading + TXT On-Demand**:
```python
# Phase 1 (Startup): Pre-load ALL YAML files
# Reason: Keys are inside file content, not file path
load_yaml_files_only()  # colors.yaml → colors, colors/warm, colors/cold

# Phase 2 (Runtime): Load TXT files on-demand
# File path = key (e.g., "flower.txt" → "__flower__")
# No metadata scan for TXT files
```

**Progressive Loading**:
```
Initial:         /list/loaded → YAML keys only (e.g., colors, colors/warm, colors/cold)
After __flower__: /list/loaded → +1 TXT wildcard
After __dragon__: /list/loaded → +2-3 (TXT transitive)
```

**⚠️ YAML Limitation**:
YAML wildcards are excluded from on-demand mode because wildcard keys exist
inside the file content. To discover `__colors/warm__`, we must parse `colors.yaml`.
Solution: Convert large YAML collections to TXT file structure for true on-demand.

#### New API Endpoint

**`GET /impact/wildcards/list/loaded`**:
```json
{
  "data": ["__colors__", "__colors/warm__", "__colors/cold__", "__samples/flower__"],
  "on_demand_mode": true,
  "total_available": 0
}
```

Note: `total_available` is 0 in on-demand mode (TXT files not pre-scanned)

**Progressive Example**:
```bash
# Initial state (YAML pre-loaded)
curl /impact/wildcards/list/loaded
→ {"data": ["__colors__", "__colors/warm__", "__colors/cold__"], "total_available": 0}

# Access first wildcard
curl -X POST /impact/wildcards -d '{"text": "__flower__", "seed": 42}'

# Check again (TXT wildcard added)
curl /impact/wildcards/list/loaded
→ {"data": ["__colors__", "__colors/warm__", "__colors/cold__", "__samples/flower__"], "total_available": 0}
```

#### Performance Improvements

**Large Dataset (10GB, 100K files)**:

| Metric | Before | After |
|--------|--------|-------|
| **Startup** | 20-60 min | **< 1 min** |
| **Memory** | 5-10 GB | **< 100MB** |
| **Size calc** | 10-30 min | **< 1 sec** |

#### Run Test

```bash
bash test_progressive_ondemand.sh
```

**Expected Output**:
```
Step 1: Initial state
  Loaded wildcards: 0

Step 2: Access __samples/flower__
  Loaded wildcards: 1
✓ PASS: Wildcard count increased

Step 3: Access __dragon__
  Loaded wildcards: 3
✓ PASS: Wildcard count increased progressively

🎉 ALL TESTS PASSED
```

---

### 2. Lazy Loading Tests

**Purpose**: Verify on-demand loading produces identical results to full cache mode.

**Test Files**:
- `test_lazy_load_api.sh` (Shell, ~3 min)
- `test_wildcard_lazy_loading.py` (Python unit test)
- `test_lazy_load_verification.py` (Python verification)

#### What's Tested

**LazyWildcardLoader Class**:
- Loads data only on first access
- Acts as list-like proxy
- Thread-safe with locking

**Mode Detection**:
- Automatic based on total size vs cache limit
- Full cache: < 50MB (default)
- On-demand: ≥ 50MB

**Consistency**:
- Full cache results == On-demand results
- Same seeds produce same outputs
- All wildcard features work identically

#### Test Scenarios

**test_lazy_load_api.sh** runs both modes and compares:

1. **Wildcard list** (before access)
2. **Simple wildcard**: `__samples/flower__`
3. **Depth 3 transitive**: `__adnd__ creature`
4. **YAML wildcard**: `__colors__`
5. **Wildcard list** (after access)

**All results must match exactly**.

#### Run Test

```bash
bash test_lazy_load_api.sh
```

**Expected Output**:
```
Testing: full_cache (limit: 100MB, port: 8190)
✓ Server started
Test 1: Get wildcard list
   Total wildcards: 1000

Testing: on_demand (limit: 1MB, port: 8191)
✓ Server started
Test 1: Get wildcard list
   Total wildcards: 1000

COMPARISON RESULTS
Test: Simple Wildcard
✓ Results MATCH

🎉 ALL TESTS PASSED
On-demand loading produces IDENTICAL results!
```

---

### 3. Sequential/Transitive Loading Tests

**Purpose**: Verify transitive wildcards expand correctly across multiple stages.

**Test Files**:
- `test_sequential_loading.sh` (Shell, ~5 min)
- `find_transitive_wildcards.sh` (Utility)

#### What's Tested

**Transitive Expansion**:
```
Depth 1: __samples/flower__ → rose
Depth 2: __dragon__ → __dragon/warrior__ → content
Depth 3: __adnd__ → __dragon__ → __dragon_spirit__ → content
```

**Maximum Depth**: 3 levels verified (system supports up to 100)

#### Test Categories

**17 tests across 5 categories**:

1. **Depth Verification** (4 tests)
   - Depth 1: Direct wildcard
   - Depth 2: One level transitive
   - Depth 3: Two levels + suffix
   - Depth 3: Maximum chain

2. **Mixed Transitive** (3 tests)
   - Dynamic selection of transitive
   - Multiple transitive in one prompt
   - Nested transitive in dynamic

3. **Complex Scenarios** (3 tests)
   - Weighted selection with transitive
   - Multi-select with transitive
   - Quantified transitive

4. **Edge Cases** (4 tests)
   - Compound grammar
   - Multiple wildcards, different depths
   - YAML wildcards (no transitive)
   - Transitive + YAML combination

5. **On-Demand Mode** (3 tests)
   - Depth 3 in on-demand
   - Complex scenario in on-demand
   - Multiple transitive in on-demand

#### Example: Depth 3 Chain

**Files**:
```
adnd.txt:
  __dragon__

dragon.txt:
  __dragon_spirit__

dragon_spirit.txt:
  Shrewd Hatchling
  Ancient Dragon
```

**Usage**:
```
__adnd__ creature
→ __dragon__ creature
→ __dragon_spirit__ creature
→ "Shrewd Hatchling creature"
```

#### Run Test

```bash
bash test_sequential_loading.sh
```

**Expected Output**:
```
=== Test 01: Depth 1 - Direct wildcard ===
Raw prompt: __samples/flower__
✓ All wildcards fully expanded
Final Output: rose
Status: ✅ SUCCESS

=== Test 04: Depth 3 - Maximum transitive chain ===
Raw prompt: __adnd__ creature
✓ All wildcards fully expanded
Final Output: Shrewd Hatchling creature
Status: ✅ SUCCESS
```

---

### 4. Versatile Prompts Tests

**Purpose**: Test all wildcard features and syntax variations.

**Test Files**:
- `test_versatile_prompts.sh` (Shell, ~2 min)
- `test_wildcard_features.sh` (Shell)
- `test_wildcard_consistency.sh` (Shell)

#### What's Tested

**30 prompts across 10 categories**:

1. **Simple Wildcards** (3 tests)
   - Basic substitution
   - Case insensitive (uppercase)
   - Case insensitive (mixed)

2. **Dynamic Prompts** (3 tests)
   - Simple: `{red|green|blue} apple`
   - Nested: `{a|{d|e|f}|c}`
   - Complex nested: `{blue apple|red {cherry|berry}}`

3. **Selection Weights** (2 tests)
   - Weighted: `{5::red|4::green|7::blue} car`
   - Multiple weighted: `{10::beautiful|5::stunning} {3::sunset|2::sunrise}`

4. **Compound Grammar** (3 tests)
   - Wildcard + dynamic: `{pencil|apple|__flower__}`
   - Complex compound: `1{girl|boy} {sitting|standing} with {__object__|item}`
   - Nested compound: `{big|small} {red {apple|cherry}|blue __flower__}`

5. **Multi-Select** (4 tests)
   - Fixed count: `{2$$, $$opt1|opt2|opt3|opt4}`
   - Range: `{2-4$$, $$opt1|opt2|opt3|opt4|opt5}`
   - With separator: `{3$$; $$a|b|c|d|e}`
   - Short form: `{-3$$, $$opt1|opt2|opt3|opt4}`

6. **Quantifiers** (2 tests)
   - Basic: `3#__wildcard__`
   - With multi-select: `{2$$, $$5#__colors__}`

7. **Wildcard Fallback** (2 tests)
   - Auto-expand: `__flower__` → `__*/flower__`
   - Wildcard patterns: `__samples/*__`

8. **YAML Wildcards** (3 tests)
   - Simple YAML: `__colors__`
   - Nested YAML: `__colors/warm__`
   - Multiple YAML: `__colors__ and __animals__`

9. **Transitive Wildcards** (4 tests)
   - Depth 2: `__dragon__`
   - Depth 3: `__adnd__`
   - Mixed depth: `__flower__ and __dragon__`
   - Dynamic transitive: `{__dragon__|__adnd__}`

10. **Real-World Scenarios** (4 tests)
    - Portrait prompt
    - Landscape prompt
    - Fantasy prompt
    - Abstract art prompt

#### Example Tests

**Test 04: Simple Dynamic Prompt**:
```
Raw: {red|green|blue} apple
Seed: 100
Result: "red apple" (deterministic)
```

**Test 09: Wildcard + Dynamic**:
```
Raw: 1girl holding {blue pencil|red apple|colorful __samples/flower__}
Seed: 100
Result: "1girl holding colorful chrysanthemum"
```

**Test 18: Multi-Select Range**:
```
Raw: {2-4$$, $$happy|sad|angry|excited|calm}
Seed: 100
Result: "happy, sad, angry" (2-4 emotions selected)
```

#### Run Test

```bash
bash test_versatile_prompts.sh
```

**Expected Output**:
```
========================================
Test 01: Basic Wildcard
========================================
Raw: __samples/flower__
Result: chrysanthemum
Status: ✅ PASS

========================================
Test 04: Simple Dynamic Prompt
========================================
Raw: {red|green|blue} apple
Result: red apple
Status: ✅ PASS

Total: 30 tests
Passed: 30
Failed: 0
```

---

## Test Execution

### Prerequisites

**Required**:
- ComfyUI installed
- Impact Pack installed
- Python 3.8+
- Bash shell
- curl (for API tests)

**Optional**:
- jq (for JSON parsing)
- git (for version control)

### Environment Setup

**1. Configure Impact Pack**:
```bash
cd /path/to/ComfyUI/custom_nodes/comfyui-impact-pack

# Create or edit config
cat > impact-pack.ini << EOF
[default]
dependency_version = 24
wildcard_cache_limit_mb = 50
custom_wildcards = $(pwd)/custom_wildcards
disable_gpu_opencv = True
EOF
```

**2. Prepare Wildcards**:
```bash
# Check wildcard files exist
ls wildcards/*.txt wildcards/*.yaml
ls custom_wildcards/*.txt
```

### Running Tests

#### Unit Tests (Python)

**Standalone** (no server required):
```bash
python3 test_wildcard_lazy_loading.py
python3 test_progressive_loading.py
```

**Note**: Requires ComfyUI environment or will show import errors.

#### Integration Tests (Shell)

**Manual Server Start**:
```bash
# Terminal 1: Start server
cd /path/to/ComfyUI
bash run.sh --listen 127.0.0.1 --port 8188

# Terminal 2: Run tests
cd custom_nodes/comfyui-impact-pack/tests
bash test_versatile_prompts.sh
```

**Automated** (tests start/stop server):
```bash
# Each test manages its own server
bash test_progressive_ondemand.sh  # Port 8195
bash test_lazy_load_api.sh         # Ports 8190-8191
bash test_sequential_loading.sh    # Port 8193
```

### Test Timing

| Test | Duration | Server | Ports |
|------|----------|--------|-------|
| `test_progressive_ondemand.sh` | ~2 min | Auto | 8195 |
| `test_lazy_load_api.sh` | ~3 min | Auto | 8190-8191 |
| `test_sequential_loading.sh` | ~5 min | Auto | 8193 |
| `test_versatile_prompts.sh` | ~2 min | Manual | 8188 |
| `test_wildcard_consistency.sh` | ~1 min | Manual | 8188 |
| Python unit tests | < 5 sec | No | N/A |

### Logs

**Server Logs**:
```bash
/tmp/progressive_test.log
/tmp/comfyui_full_cache.log
/tmp/comfyui_on_demand.log
/tmp/sequential_test.log
```

**Check Logs**:
```bash
# View recent wildcard logs
tail -50 /tmp/progressive_test.log | grep -i wildcard

# Find errors
grep -i "error\|fail" /tmp/*.log

# Check mode activation
grep -i "mode" /tmp/progressive_test.log
```

---

## Expected Results

### Success Criteria

#### Progressive Loading
- ✅ `/list/loaded` starts at 0 (or low count)
- ✅ `/list/loaded` increases after each unique wildcard
- ✅ `/list/loaded` unchanged on cache hits
- ✅ Transitive wildcards load multiple entries
- ✅ Final results identical to full cache mode

#### Lazy Loading
- ✅ Full cache results == On-demand results (all tests)
- ✅ Mode detection correct (based on size vs limit)
- ✅ LazyWildcardLoader loads only on access
- ✅ All API endpoints return consistent data

#### Sequential Loading
- ✅ Depth 1-3 expand correctly
- ✅ Complex scenarios work (weighted, multi-select, etc.)
- ✅ On-demand mode matches full cache
- ✅ No infinite loops (max 100 iterations)

#### Versatile Prompts
- ✅ All 30 test prompts process successfully
- ✅ Deterministic (same seed → same result)
- ✅ No syntax errors
- ✅ Proper probability distribution

### Sample Output

**Progressive Loading Success**:
```
========================================
Progressive Loading Verification
========================================

Step 1: Initial state
  On-demand mode: True
  Total available: 1000
  Loaded wildcards: 0

Step 2: Access __samples/flower__
  Result: rose
  Loaded wildcards: 1
✓ PASS

Step 3: Access __dragon__
  Result: ancient dragon
  Loaded wildcards: 3
✓ PASS

🎉 ALL TESTS PASSED
Progressive on-demand loading verified!
```

**Lazy Loading Success**:
```
========================================
COMPARISON RESULTS
========================================

Test: Wildcard List (before)
✓ Results MATCH

Test: Simple Wildcard
✓ Results MATCH

Test: Depth 3 Transitive
✓ Results MATCH

🎉 ALL TESTS PASSED
On-demand produces IDENTICAL results!
```

---

## Troubleshooting

### Common Issues

#### 1. Server Fails to Start

**Symptoms**:
```
✗ Server failed to start
curl: (7) Failed to connect
```

**Solutions**:
```bash
# Check if port in use
lsof -i :8188
netstat -tlnp | grep 8188

# Kill existing processes
pkill -f "python.*main.py"

# Increase startup wait time
# In test script: sleep 15 → sleep 30
```

#### 2. Module Not Found (Python)

**Symptoms**:
```
ModuleNotFoundError: No module named 'modules'
```

**Solutions**:
```bash
# Option 1: Run from ComfyUI directory
cd /path/to/ComfyUI
python3 custom_nodes/comfyui-impact-pack/tests/test_progressive_loading.py

# Option 2: Add to PYTHONPATH
export PYTHONPATH=/path/to/ComfyUI/custom_nodes/comfyui-impact-pack:$PYTHONPATH
python3 test_progressive_loading.py
```

#### 3. On-Demand Mode Not Activating

**Symptoms**:
```
Using full cache mode.
```

**Check**:
```bash
# View total size
grep "Wildcard total size" /tmp/progressive_test.log

# Check cache limit
grep "cache_limit_mb" impact-pack.ini
```

**Solutions**:
```bash
# Force on-demand mode
cat > impact-pack.ini << EOF
[default]
wildcard_cache_limit_mb = 0.5
EOF
```

#### 4. Tests Timeout

**Symptoms**:
```
Waiting for server startup...
✗ Server failed to start
```

**Solutions**:
```bash
# Check system resources
free -h
df -h

# View server logs
tail -100 /tmp/progressive_test.log

# Manually test server
cd /path/to/ComfyUI
bash run.sh --port 8195

# Increase timeout in test
# sleep 15 → sleep 60
```

#### 5. Results Don't Match

**Symptoms**:
```
✗ Results DIFFER
```

**Debug**:
```bash
# Compare results
diff /tmp/result_full_cache_simple.json /tmp/result_on_demand_simple.json

# Check seeds are same
grep "seed" /tmp/result_*.json

# Verify same wildcard files used
ls -la wildcards/samples/flower.txt
```

**File Bug Report**:
- Wildcard text
- Seed value
- Full cache result
- On-demand result
- Server logs

#### 6. Slow Performance

**Symptoms**:
- Tests take much longer than expected
- Server startup > 2 minutes

**Check**:
```bash
# Wildcard size
du -sh wildcards/

# Disk I/O
iostat -x 1 5

# System resources
top
```

**Solutions**:
- Use SSD (not HDD)
- Reduce wildcard size
- Increase cache limit (use full cache mode)
- Close other applications

---

## Performance Benchmarks

### Expected Performance

**Small Dataset (< 50MB)**:
```
Mode: Full cache
Startup: < 10 seconds
Memory: ~50MB
First access: Instant
```

**Medium Dataset (50MB - 1GB)**:
```
Mode: On-demand
Startup: < 30 seconds
Memory: < 200MB initial
First access: 10-50ms per wildcard
```

**Large Dataset (10GB+)**:
```
Mode: On-demand
Startup: < 1 minute
Memory: < 100MB initial
First access: 10-50ms per wildcard
Memory growth: Progressive
```

### Optimization Tips

**For Faster Tests**:
1. Use smaller wildcard dataset
2. Run specific tests (not all)
3. Use manual server (keep running)
4. Skip sleep times (if server already running)

**For Large Datasets**:
1. Verify on-demand mode activates
2. Monitor `/list/loaded` to track memory
3. Use SSD for file storage
4. Organize wildcards into subdirectories

---

## Contributing

### Adding New Tests

**1. Create Test File**:
```bash
touch tests/test_new_feature.sh
chmod +x tests/test_new_feature.sh
```

**2. Test Template**:
```bash
#!/bin/bash
# Test: New Feature
# Purpose: Verify new feature works correctly

set -e

PORT=8XXX
IMPACT_DIR="/path/to/comfyui-impact-pack"

# Setup config
cat > impact-pack.ini << EOF
[default]
wildcard_cache_limit_mb = 50
EOF

# Start server
cd /path/to/ComfyUI
bash run.sh --port $PORT > /tmp/test_new.log 2>&1 &
sleep 15

# Test
RESULT=$(curl -s http://127.0.0.1:$PORT/impact/wildcards/list)

# Validate
if [ "$RESULT" = "expected" ]; then
    echo "✅ PASS"
    exit 0
else
    echo "❌ FAIL"
    exit 1
fi
```

**3. Update Documentation**:
- Add test description to this README
- Update test count
- Add to appropriate category

### Testing Guidelines

**Test Structure**:
1. Clear purpose statement
2. Setup (config, wildcards)
3. Execution (API calls, processing)
4. Validation (assertions, comparisons)
5. Cleanup (kill servers, restore config)

**Good Practices**:
- Use unique port numbers
- Clean up background processes
- Provide clear success/failure messages
- Log to `/tmp/` for debugging
- Use deterministic seeds
- Test both modes (full cache + on-demand)

---

## Reference

### Test Files Quick Reference

```bash
# Progressive loading
test_progressive_ondemand.sh          # Integration test
test_progressive_loading.py           # Unit test

# Lazy loading
test_lazy_load_api.sh                 # Integration test
test_wildcard_lazy_loading.py         # Unit test

# Sequential/transitive
test_sequential_loading.sh            # Integration test
find_transitive_wildcards.sh          # Utility

# Features
test_versatile_prompts.sh             # Comprehensive features
test_wildcard_features.sh             # Core features
test_wildcard_consistency.sh          # Consistency

# Validation
test_wildcard_final.py                # Final validation
test_lazy_load_verification.py        # Lazy load verification
```

### Documentation

- **System Overview**: `../docs/WILDCARD_SYSTEM_OVERVIEW.md`
- **Testing Guide**: `../docs/WILDCARD_TESTING_GUIDE.md`

### API Endpoints

```
GET  /impact/wildcards/list           # All available wildcards
GET  /impact/wildcards/list/loaded    # Actually loaded (progressive)
POST /impact/wildcards                # Process wildcard text
GET  /impact/wildcards/refresh        # Reload all wildcards
```

---

**Last Updated**: 2024-11-17
**Total Tests**: 11 files, 100+ scenarios
**Coverage**: ~95% of wildcard features