Over the past two weeks, we conducted a brief interview with the outstanding developers participating in this year's Summer of Open Source program, gaining an initial understanding of their development process and takeaways. Today, through their full final reports, we will take a deeper look into the technical details of each project, with the hope of helping everyone better understand the latest progress in the Apache DolphinScheduler project.
Below is the complete report for the project “Adding a gRPC Plugin to Apache DolphinScheduler”:
1. Project Objective
DolphinScheduler is an open-source distributed workflow scheduling system. To meet users’ needs for invoking gRPC services within workflows, a gRPC plugin for DolphinScheduler was designed and implemented.
This plugin aims to:
- Support flexible configuration of gRPC service address, port, method name, request parameters, and other information in task settings.
- Provide both a visual interface and JSON-based configuration for specifying gRPC request parameters.
- Ensure that task execution results can be correctly returned and bound to the DolphinScheduler workflow context.
- Support exception handling and retry mechanisms to improve task stability.
- Implement SSL/TLS-encrypted connections to ensure data security.
2. Task Implementation
According to the RFC document, the plugin fulfills the following requirements:
Configuration is similar to the HTTP Task, reducing the learning curve for users.
Functional Requirements
2.1 Frontend
- Added gRPC task–related files and icons.
- Implemented the parameter form, including configuration items for service address, port, gRPC service definition, request method, request content, parameter list, and more.
- Used protobuf.js to implement JSON-to-Protobuf type mapping.
2.2 Backend
- Created foundational code for gRPC Task and implemented timeout and retry features.
- Parsed parameters received from the frontend and supported dynamic gRPC invocation generation.
- Implemented error handling and workflow context binding, with validation.
- Developed request sending, response processing, exception capturing, and retry logic.
- Security: Supported establishing connections using client-side SSL/TLS.
2.3 Test Coverage
- Comprehensive unit tests for overall plugin functionality:
- https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-grpc/src/test/java/org/apache/dolphinscheduler/plugin/task/grpc/GrpcTaskTest.java
- Basic gRPC functionality tests
- Workflow and context binding tests
- Exception handling tests
- Unit tests for Protobuf parser syntax support:
- https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-grpc/src/test/java/org/apache/dolphinscheduler/plugin/task/grpc/GrpcParserTest.java
- Basic type tests
- Tests for nested types such as map and repeated fields
- Tests for optional and oneof fields
- API tests based on task parameters:
- https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-api-test/dolphinscheduler-api-test-case/src/test/java/org/apache/dolphinscheduler/api/test/cases/tasks/GrpcTaskAPITest.java
2.4 Documentation & Examples
- gRPC Task Plugin—Chinese documentation and examples:
- https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/zh/guide/task/grpc.md
- gRPC Task Plugin—English documentation and examples:
- https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/guide/task/grpc.md
3. Summary of Results
- Completed the design and development of the DolphinScheduler gRPC plugin.
- Completed unit and integration testing, covering major functionalities and edge cases.
- Provided detailed plugin usage documentation, parameter lists, and examples.
Project GitHub link:
https://github.com/apache/dolphinscheduler/pull/17402
We express our gratitude to OSPP and all DolphinScheduler community mentors for their support and guidance.