Posts
-
Apache Nifi Record Path Cheat Sheet
Apache Nifi Record path allows dynmic values in functional fields, and manipulation of a record as it is passing through Nifi and heavily used in the UpdateRecord and ConvertRecord processors. This is a short reference to find useful functions and examples. For a full reference see the offical documentation.
-
Auto Loading Nars in Apache Nifi
Apache Nifi, as of 1.9.0, now allows live reloading for nars. This was a major pain point in providing custom processors to a live cluster and keeping processes in line with active code. Now it is as simple as placing the nar in the $NIFI_HOME/extensions directory.
-
Apache Nifi Expression Language Cheat Sheet
Apache Nifi Expression language allows dynmic values in functional fields. This is a short reference to find useful functions and examples. For a full reference see the offical documentation.
-
Running a cluster with Apache Nifi and Docker
If you don’t want the details behind Nifi’s clustering, you can skip ahead to running a cluster.
Clustering Apache nifi has multiple benefits that come along with the complications it introduces, the main benefit being more throughput and data processing power. If you’ve run a single machine instance of nifi before, you know that having data intense processors running can take a hit on all of your resources depending on the type of tasks you are doing. We’ve maxed CPU with CSV/JSON parsing, IO with creating hundreds of thousands of records from gigabyte+ file ingestions, and RAM with large XML tree traversal that never seems avoidable. There are other ways to try and solve some of these resource issues (RAID, spanning repositories across disks, flow optimizations etc.), but one of the quickest ones is scale out horizontally, create a nifi cluster! With Nifi’s clustering, these tasks become more managable and you won’t look at your system stats wondering if everything will be ok… and it will, because we are here to help!
-
Working with the Official NiFi Docker Image
Updates as of January 29th, 2019 - Nifi v1.8.0
Nifi’s official docker image has come quite a ways since the first release in 1.2.0. There have been some quality of life changes as well as fixes to things missed on the first run through. We are updating this post for most of those new additions and have a full post about clustering in docker. The readme/quickstart on Nifi’s docker hub page has been fleshed out and contains quite a bit of documentation, so definitely check there for more features.
To start with, there is now a latest tag, so a quick
docker pull apache/nifinow works! Quite a few of the nifi.properties have been exposed via environment variables and you can now run a secured instance, via certificates or LDAP, through the official image. This includes clustering variables such as address, protocol, max threads, zookeeper address, etc. The docker image also now contains the nifi-toolkit, which allows you to manage a cluster and run various commands from the cli. To use the toolikt, after the container is running, you can just run a docker exec command,docker exec -ti nifi nifi-toolkit-current/bin/cli.sh nifi current-user, to get the current user, which should print outanonymous. We’ll go into a deeper dive about using the official docker image in our clustering post. -
Monitoring Nifi with Data Dog
Monitoring your nifi deployment is almost as important as setting it all up. Knowing that your flow is performing as expected and that the data isn’t throwing errors, blocking queues, or causing other problems requires some ongoing tasks, be it daily checking nifi, or some sort of monitoring. Nifi has some built in reporting tasks which can be used for this, and one of them is a Datadog Reporting Task. This reports 25 different metrics about nifi and the JVM. Ranging from queued bytes to JVM garbage collection, plus more for each processor you have running in your flow, the overall stats are pretty useful. This post will walk through setting this up form start to finish.
-
Developing a Custom Apache Nifi Reporting Task
Reporting tasks are background tasks commonly responsible for communicating system’s state, flow metrics, and other information commonly used for monitoring and alerting. The current list of reporting tasks cover a great deal of information and generally available as site to site reporting tasks. This generally means sending the data back into a flow to filter and transform before sending to its destination. While this works incredibly well, if the flow would require a custom processor, in its place you could develop a custom reporting task. The full source is hosted on Github or Gitlab.
-
Apache Nifi Release 1.4.0 Highlights
Apache Nifi released version 1.4.0 in October 2017. I know we are late to talk about the release highlights, but better late than never. This will just be a quick run down for those interested or curious if they should upgrade their cluster from a previous release to the new and shiny! As usual, the release notes are on nifi’s confluence page, the issues resolved are on their jira and you can download the newest, and previous releases from the nifi site.
There were 204 issues closed in total.
- Bug Fixes: 97
- Improvements]: 87
- New Features: 14
There were a few other tasks and subtasks that were resolved that make up the other 6 items.
-
Apache Nifi Processors Versions - Documentation
The other day I was using an older version of Apache Nifi than the most currently released and realized that the only way I could access the processor docks was via my locally running verion. You can open up the docs for the version you are on via the right click menu on a processor and even pop it out to a new tab or window, and while this is nice, I wanted a quick rundown or searchable area outside of the environment that I could use to lookup processors in. That way I wasn’t tied down to my instance - say I wanted to look at it on the go on my phone or whatever the use case may be. This spawned us to start doing something we had been talking about for quite some time - versioning our apache nifi processors page. We now have a nice dropdown on all processors pages that allows you to select from the most recent 0.x line as of writing this, 0.7.4, and the 1.x line from 1.2 on - 1.2, 1.3 and 1.4. Example dropdown:
Includes all processors through releaseWe think this is a pretty handy feature and hope that you all find it useful too! If you have any other thoughts or feedback about useful tools or posts, feel free to let us know at info@nifi.rocks.
-
Apache Nifi Release 1.2.0 Highlights
Apache Nifi’s latest release is 1.2.0. I know we skipped the whole 1.0.0 release highglights, so a quick breakdown of overall changes follows for the jump from 0.* to 1.*, since there were quite a few major breaking changes. For 1.2.0, there were 381 issues closed or resolved, with a break down of issues to follow. Apache Nifi 1.2.0 can be downloaded from Apache here, full release notes can be found on their jira, and Highlights of the release on their Confluence page.
- Bug Fixes: 212
- Improvements: 129
- New Features: 24
There were a few other tasks and subtasks that were resolved that make up the other 16 items, but since they were pretty basic - updating Jetty and jQuery, we’ll probably skip going over most of them (except for the official Docker image!)
subscribe via RSS