• Monitoring Nifi with Data Dog

    Monitoring your nifi deployment is almost as important as setting it all up. Knowing that your flow is performing as expected and that the data isn’t throwing errors, blocking queues, or causing other problems requires some ongoing tasks, be it daily checking nifi, or some sort of monitoring. Nifi has some built in reporting tasks which can be used for this, and one of them is a Datadog Reporting Task. This reports 25 different metrics about nifi and the JVM. Ranging from queued bytes to JVM garbage collection, plus more for each processor you have running in your flow, the overall stats are pretty useful. This post will walk through setting this up form start to finish.

    Read More →

  • Developing a Custom Apache Nifi Reporting Task

    Reporting tasks are background tasks commonly responsible for communicating system’s state, flow metrics, and other information commonly used for monitoring and alerting. The current list of reporting tasks cover a great deal of information and generally available as site to site reporting tasks. This generally means sending the data back into a flow to filter and transform before sending to its destination. While this works incredibly well, if the flow would require a custom processor, in its place you could develop a custom reporting task. The full source is hosted on Github or Gitlab.

    Read More →

  • Apache Nifi Release 1.4.0 Highlights

    Apache Nifi released version 1.4.0 in October 2017. I know we are late to talk about the release highlights, but better late than never. This will just be a quick run down for those interested or curious if they should upgrade their cluster from a previous release to the new and shiny! As usual, the release notes are on nifi’s confluence page, the issues resolved are on their jira and you can download the newest, and previous releases from the nifi site.

    There were 204 issues closed in total.

    There were a few other tasks and subtasks that were resolved that make up the other 6 items.

    Read More →

  • Apache Nifi Processors Versions - Documentation

    The other day I was using an older version of Apache Nifi than the most currently released and realized that the only way I could access the processor docks was via my locally running verion. You can open up the docs for the version you are on via the right click menu on a processor and even pop it out to a new tab or window, and while this is nice, I wanted a quick rundown or searchable area outside of the environment that I could use to lookup processors in. That way I wasn’t tied down to my instance - say I wanted to look at it on the go on my phone or whatever the use case may be. This spawned us to start doing something we had been talking about for quite some time - versioning our apache nifi processors page. We now have a nice dropdown on all processors pages that allows you to select from the most recent 0.x line as of writing this, 0.7.4, and the 1.x line from 1.2 on - 1.2, 1.3 and 1.4. Example dropdown:

    Includes all processors through release

    We think this is a pretty handy feature and hope that you all find it useful too! If you have any other thoughts or feedback about useful tools or posts, feel free to let us know at

    Read More →

  • Working with the Official NiFi Docker Image

    Apache Nifi’s latest release, 1.2.0, brought with it an official docker image on the docker hub. So let’s get started.

    Pull the docker image, note after a latest tag is created you can drop the release version.

    docker pull apache/nifi:1.2.0

    Next, start the image as is to see it run. The -p 8080:8080 exposes the remote nifi port locally.

    docker run -p 8080:8080 apache/nifi:1.2.0

    Read More →

  • Apache Nifi Release 1.2.0 Highlights

    Apache Nifi’s latest release is 1.2.0. I know we skipped the whole 1.0.0 release highglights, so a quick breakdown of overall changes follows for the jump from 0.* to 1.*, since there were quite a few major breaking changes. For 1.2.0, there were 381 issues closed or resolved, with a break down of issues to follow. Apache Nifi 1.2.0 can be downloaded from Apache here, full release notes can be found on their jira, and Highlights of the release on their Confluence page.

    There were a few other tasks and subtasks that were resolved that make up the other 16 items, but since they were pretty basic - updating Jetty and jQuery, we’ll probably skip going over most of them (except for the official Docker image!)

    Read More →

  • Getting Familiar: the ExecuteScript Processor - Groovy

    I came across a question on the nifi dev mailing list and thought it would make a good example solving a real world problem, building off of our previous ExecuteScript post. As a side note, since Elasticsearch uses json for their documents and the PutElasticsearch processors expect the flow file to be json, you could use the EvaluateJsonPath Processor to put the field you want as an attribute.

    Read More →

  • Getting Familiar: The ExecuteScript Processor

    We’ve decided to start a new series for getting familiar with the different Apache Nifi services and processors, and I’m calling it “Getting Familiar”. We’ll try to post fairly often about different processors, using the controller services and configuring certain things in Nifi.The first one in the series will be about the ExecuteScript processor.

    Read More →

  • Apache Nifi Release 0.7.0 Highlights

    Apache Nifi’s newest release is out, 0.7.0. You can grab the binaries from their site as always. So lets dive in and see what to look for in this release!

    As always, a bunch of bug fixes, but this time there are quite a few improvements.

    Read More →

  • Apache Nifi Release 0.6.0 Highlights

    Apache Nifi release 0.6.0 and 0.6.1 recently and I wanted to go throught and highlight some of the key changes that may affect you Nifi users. So far Nifi has kept pretty well to thier 6 week release schedule that they have discussed. That’s a really agressive release schedule so we’ll see if they can keep that up once they hit the 0.7.0/1.0.0 release, which they say is going to actually be pushed back a little. The 0.6.0 release was again substantial with 52 bug fixes, 15 improvements and 7 new features. 0.6.1 didn’t take to long to follow with 11 bug fixes and 1 improvement. The release binaries are on the Apache Nifi download pageThe full release notes are avaialble on Apaches Nifi’s wiki page. The combined 0.6.* release information is below.

    Read More →

subscribe via RSS