DataJuicer Agent

​

DataJuicer Agent

​

Why DataJuicer Agent?

​

What Does This Agent Do?

• Intelligent Query: Match operators from ~200 options via natural language
• Automated Pipeline: Generate and run Data-Juicer YAML from descriptions
• Custom Extension: Develop and integrate custom operators locally

​

Architecture

​

Multi-Agent Routing Architecture

User Query
    ↓
Router Agent (Task Triage)
    ├── Standard Data Processing Task → Data Processing Agent (DJ Agent)
    │   ├── Preview data samples (confirm field names and data format)
    │   ├── query_dj_operators (semantic matching of operators)
    │   ├── Generate YAML configuration file
    │   └── execute_safe_command (execute dj-process, dj-analyze)
    │
    └── Custom Operator Development → Code Development Agent (DJ Dev Agent)
        ├── `get_basic_files` (get base classes and registration mechanism)
        ├── `get_operator_example` (get similar operator examples)
        ├── Generate code compliant with specifications
        └── Local integration (register to user-specified path)

​

Two Integration Modes

• Tool Binding: Calls CLI (dj-analyze, dj-process); low migration cost.
• MCP Binding: Calls Data-Juicer MCP directly; no intermediate YAML, better performance.

​

Quick Start

uv pip install -r requirements.txt   # or: pip install -r requirements.txt

export DASHSCOPE_API_KEY="your-dashscope-key"
export DATA_JUICER_PATH="your-data-juicer-path"   # optional, for custom ops

# Use AgentScope Studio's interactive interface (please install and start AgentScope Studio first)
python main.py --use_studio True

# Or use command line mode directly (default)
python main.py

npm install -g @agentscope/studio
as_studio

​

Agent Introduction

​

Data Processing Agent

1. Data Preview: Preview the first 5–10 data samples to confirm field names and data format—this is a crucial step to avoid configuration errors
2. Operator Retrieval: Call the query_dj_operators tool to semantically match suitable operators
3. Parameter Decision: LLM autonomously decides global parameters (such as dataset_path, export_path) and specific operator configurations
4. Configuration Generation: Generate standard YAML configuration files
5. Execute Processing: Call the dj-process command to execute actual processing

• Data Cleaning: Deduplication, removal of low-quality samples, format standardization
• Multimodal Processing: Process text, image, and video data simultaneously
• Batch Conversion: Format conversion, data augmentation, feature extraction

View Complete Example Log (from AgentScope Studio)

1. Call query_dj_operators, accurately returning two operators: text_length_filter and image_size_filter
2. Use view_text_file tool to preview raw data, confirming fields are indeed ‘text’ and ‘image’
3. Generate YAML configuration and save to temporary path via write_text_file
4. Call execute_safe_command to execute dj-process, returning result path

​

Code Development Agent (DJ Dev Agent)

1. Operator Retrieval: Find existing operators with similar functionality as references
2. Get Templates: Pull base class files and typical examples to ensure consistent code style
3. Generate Code: Based on the function prototype provided by the user, generate operator classes compliant with DataJuicer specifications
4. Local Integration: Register the new operator to the user-specified local codebase path

• Implement Operator: Create operator class file, inherit from Mapper/Filter base class, register using @OPERATORS.register_module decorator
• Update Registration: Modify __init__.py, add new class to __all__ list
• Write Tests: Generate unit tests covering multiple scenarios, including edge cases, ensuring robustness

• Develop domain-specific filter or transformation operators
• Integrate proprietary data processing logic
• Extend Data-Juicer capabilities for specific scenarios

View Complete Example Log (from AgentScope Studio)

​

Advanced Features

​

Operator Retrieval

• LLM (default): Qwen-Turbo semantic match; best accuracy, higher tokens.
• Vector (vector): DashScope embedding + FAISS; fast, lower cost.
• Auto (auto): LLM first, fallback to vector.

python main.py --retrieve_mode vector

​

MCP Agent

​

MCP Server Types

• Provides two tools: get_data_processing_ops and run_data_recipe
• Retrieves by operator type, applicable modalities, and other tags, no need to call LLM or vector models
• Suitable for standardized, high-frequency scenarios with better performance

• Wraps each built-in operator as an independent tool, runs on call
• Returns all operators by default, but can control visible scope through environment variables
• Suitable for fine-grained control, building fully customized data processing pipelines

​

Configuration

{
    "mcpServers": {
        "DJ_recipe_flow": {
            "url": "http://127.0.0.1:8080/sse"
        }
    }
}

​

Usage Methods

# Enable MCP Agent and Dev Agent
python main.py --available_agents [dj_mcp,dj_dev]

# Or use shorthand
python main.py -a [dj_mcp,dj_dev]

​

Customization and Extension

​

Custom Prompts

​

Model Replacement

• Main Agent uses qwen-max for complex reasoning
• Development Agent uses qwen3-coder-480b-a35b-instruct to optimize code generation quality

​

Extending New Agents

​

Roadmap

​

Data-Juicer Q&A Agent

​

Interactive Data Analysis and Visualization Agent (In Development)

• Users can view statistics, attribution analysis, and visualization results
• Dynamically edit recipes, approve or reject suggestions
• Underpinned by dj.analyzer (data analysis), dj.attributor (effect attribution), and dj.sandbox (experiment management)
• Supports closed-loop optimization based on validation tasks

​

Other Directions

• Data Processing Agent Benchmarking: Quantify the performance of different Agents in terms of accuracy, efficiency, and robustness
• Data “Health Check Report” & Data Intelligent Recommendation: Automatically diagnose data problems and recommend optimization solutions
• Router Agent Enhancement: More seamless, e.g., when operators are lacking → Code Development Agent → Data Processing Agent
• MCP Further Optimization: Embedded LLM, users can directly use MCP connected to their local environment (e.g., IDE) to get an experience similar to current data processing agents
• Knowledge Base and RAG-oriented Data Agents
• Better Automatic Processing Solution Generation: Less token usage, more efficient, higher quality processing results
• Data Workflow Template Reuse and Automatic Tuning: Based on DataJuicer community data recipes
• …

​

Common Issues

​

Optimization Recommendations

• For large-scale data processing, it is recommended to use DataJuicer’s distributed mode
• Set batch size appropriately to balance memory usage and processing speed
• For more advanced data processing features (synthesis, Data-Model Co-Development), please refer to DataJuicer documentation

​

Related Resources