builder injector dart prevent creating empty files

I am trying to follow an example and use Dart inject in my source.

this is my inject_generator.build.yaml

# Read about `build.yaml` at https://pub.dartlang.org/packages/build_config
targets:
  $default:
    builders:
      inject_generator:
        enabled: true
    sources:
      - "lib/src/injectors"
builders:
  inject_generator:
    target: ":inject_generator"
    import: "package:inject_generator/inject_generator.dart"
    builder_factories:
      - "summarizeBuilder"
      - "generateBuilder"
    build_extensions:
      ".dart":
        - ".inject.summary"
        - ".inject.dart"
    auto_apply: dependents
    build_to: source

It works, but it generates empty files for all my files. I think it only needs to generate file for my injector - unless I am wrong.

All my attempts to prevent builder from generating empty files failed. How do I control which files build_runner scans and generates?