Apache Nifi graduated from a podling to a top level Apache Software Foundation project a few months ago, which is great news! Congratulations to the NiFi team and all the contributors to making Apache Nifi great and meeting the requirements to move from an incubating project to a top level project.
The Apache Nifi framework has built in unit testing with Junit using test runners. They have a few examples in their code base, but learning first hand really helps. In this post we’ll go over adding unit tests to the JSON Processor that we developed previously.
To start, we’ll checkout the JSON Processor code from Github and then open it up in your favorite text editor. There is already a test package/folder in the project that contains a unit test, rocks.nifi.examples.processors/JsonProcessorTest.java.
When unit testing in Apache Nifi, there are a few items on top of the normal JUnit annotations that are required. While you can unit test with just JUnit, using the built in method in Apache Nifi makes it much easier. In a future post we’ll show you how to unit test using Mockito and JUnit to test helper methods where actually invoking a full processor seems excessive, or if you just don’t want to use Nifi’s test runner.
Apache Nifi just kicked out their second release, 0.0.2. It doesn’t have too many new features in it, hence a patch release according to semantic versioning, but it is definitely an improvement over the previous version, with it’s bug fixes, improvements, and new features numbers below.
The list of available Apache Nifi processors is extensive, as documented in this post. There is still a need to develop your own; to pull data from a database, to process an uncommon file format, or many other unique situations. So to get you started, we will work through a basic processor that takes a json file as input and a json path as a parameter to place into the contents and an attribute. The full source is hosted on Github.
Includes all processors through release 1.2.0
I looked around at what can be done with Apache NiFi and didn’t notice a list of processors without looking at the code or building the project. I think a list of available processors, the work horse of Apache Nifi, would greatly help decide if it is right for certain needs. So, I went into the usage guide in the Apache Nifi UI and pulled a list of processors and a quick description for those who want to know what possibilities there are before getting into nifi itself!
List of processors
With new releases of Nifi, the number of processors have increased from the original 53 to 154 to what we currently have today! Here is a list of all processors, listed alphabetically, that are currently in Apache Nifi as of the most recent release. Each one links to a description of the processor further down. The Usage documentation available in the web ui has much more detail about each processor, it’s properties, modifiable attributes, and relationships and each processor has it’s own page in the UI, so here is just a quick overview. Again, this content is taken directly from Nifi’s Usage guide in their web UI and all credit/rights belong to them under the Apache 2.0 License.
Nifi has improved their documentation, which was originally only available when running apache nifi. The documentation now is produced through the build process and has been added to their website. So if you need more information or more detail about each processor just check there.
Apache Nifi just had their first release, 0.0.1. It showed good movement forward with 75 bug fixes, 24 improvements, and 2 new features. A list of Release Notes is on their Jira page. With the release, it gives the users source downloads to start from, which is great, but lets look at the highlights of this release.
Apache Nifi Release 0.0.1 Highlights
- One of the biggest changes was the directory structure to allow parrallel maven builds. This took the build time down from 20-30 minutes to 3-7 minutes. Nifi Jira 169
- The improvement of the assembly process also allowed for more stream lined packaging. Nifi Jira 228
Outside of the build process, most of the updates were cleaning up the code base to make it more developer friendly and fixing alot of small bugs;
- Updating libraries to their most current version
- Adding documentation/users guides
- Guarantee builds on varying OSes: OSX, Linux, Windows.
I’m pretty excited to see what’s in the next release, but also excited that I don’t have to build nifi to use it. Release binaries are available on the nifi website.
The main user interface of Apache Nifi is their web ui. This makes it much more enjoyable to use than a command line interface, but can still be hard to grasp quickly or know where certain things are when you first start using it. To help reduce the learning curve, we are going through and breaking down the web ui through a video showing how-to navigate Apache Nifi.
Our main points will be the menu bar and building a data flow with some tips and tricks along the way. The data flow will be based off of our previous post Getting Started with Apache Nifi.
- Update April 8th 2015: The Vagrantfile has been updated to pull the release tag 0.0.2 for stability
Apache NiFi is currently in incubation and so does not have any releases, so to start we have to checkout the project and build the code base. A users guide is avaialble on the nifi website with requirements for building and running nifi, mainly java 7 and maven 3.0.5+. OpendJDK will work but currently the unit tests will not pass. A quick tldr; can be found below.
If you like to keep your project dependencies seperate like me, I’ve created a Vagrantfile that will get all dependencies and then build the project in the shared /vagrant/ folder. After building it will start nifi and you can point your browser to localhost:8080/nifi/ and skip to Building a Simple Dataflow. The provisioning of the VM will take about 10-15 minutes depending on your hardware.
subscribe via RSS