Uploaded image for project: 'COmanage'
  1. COmanage
  2. CO-2668

ApiSource Kafka - Unreported Kafka Errors

    XMLWordPrintable

Details

    Description

      This report pertains to a known issue with COManage recording failed Kafka connections in ApiSource poll jobs as complete with 0 records.  We have been reviewing the php-rdkafka library, and we're able to register an error callback handler, trigger librdkafka to fire callbacks by issuing a consumer poll after each topic consumeBatch call, and bubble errors from the callback handler to the COmanage job.

      For your consideration, I've attached a git patch file for KafkaServer.php that registers an error handler, aggregates all received error messages in a class-level array, and raises an exception with its message set to a concatenation of values in that array.  So far we haven't collected an aggregate error message longer than the 256 cm_co_jobs.finish_summary field character limit; increasing the length of this field may or may not make sense.

       

      Attachments

        Activity

          People

            benn.oshrin@at.internet2.edu Benn Oshrin (internet2.edu)
            trevor.buchanan@at.internet2.edu Trevor Buchanan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: